JP7705451B2 - Cradle for creating color clouds - Google Patents

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Patent Application No. 17/107,073, filed November 30, 2020, and French Patent Application No. 2101813, filed February 25, 2021, the entire contents of each of which are incorporated herein by reference.

）に限定され、検出のためのこの閾値は、ヒトによる毛髪の色においてまだ決定されていない。更に、単一の色（例えば、ＲＧＢ又はＬ＊ａ＊ｂ＊）は測定領域の平均色であり、平均色は具体的ではなく、むしろ数学的概念である。理論的には、例えば、毛髪の領域の平均的な色を取り、その色は毛髪のその領域に存在さえしない場合がある。 Spectrophotometers are used to read the color of hair swatches, but provide only a single digital representation of the material in the form of a single RGB, L*a*b*, LCh or HSV triplet, and typically only read an area of less than 1 cm x 1 cm. Furthermore, when comparing two materials, the analysis is based on the Euclidean distance in color space between the three spaces (e.g. delta-E1976=

), and this threshold for detection has yet to be determined in human hair color. Furthermore, the single color (e.g., RGB or L*a*b*) is the average color of the measured area, and average color is not concrete, but rather a mathematical concept. In theory, one could, for example, take the average color of an area of hair, and that color may not even be present in that area of hair.

The present disclosure is directed to an apparatus for creating a color cloud, the apparatus comprising: a cradle having a first end, a second end having a window, and at least one wall connecting the first end to the second end; and a video recording device at the first end of the cradle, the video recording device configured to capture a video of a material viewed through the window. In another aspect, the present disclosure is also directed to a system for creating a color cloud of an object, the system comprising: a cradle having a first end, a second end having a window, and at least one wall connecting the first end to the second end; an image sensor attached to the first end of the cradle, the image sensor configured to capture a plurality of digital images of the object viewed through the window; and a processing circuit operably coupled to the image sensor, the processing circuit configured to generate a three-dimensional virtual representation of color information associated with the plurality of digital images of the object, and to identify one or more significant characteristics of the object and generate one or more virtual instances on a graphical user interface display that indicate the identity of the significant characteristics of the object based on one or more inputs associated with the three-dimensional virtual representation of the color information of the object.

In one embodiment, the video recording device is a camera on a mobile device.

In one embodiment, the device further comprises a light source attached to at least one of the first end, the second end, and the at least one wall.

In one embodiment, the light source is a flashlight on the mobile device.

In one embodiment, the apparatus further includes a processing circuit connected to the video recording device and configured to create a color cloud from the video of the material.

In one embodiment, the apparatus further comprises a holder attached to the first end of the cradle for holding the video recording device.

In one embodiment, the second end is colored with multiple colors adjacent the window for color separation.

It shows a user dragging a cradle over hair to capture video and create a color cloud for it. 1 illustrates one way to create and use a color cloud. 1 shows a first embodiment of a cradle. 2 shows a second embodiment of the cradle, which can be attached to a smartphone. A user's hair is shown that was used to capture a video, the video being captured over the length of the user's hair (starting at the root of the hair and ending at the tip of the hair). An example of multiple frames of video laid out in chronological order is shown, where a user captures a video of their hair starting from the roots and ending at the tips of the hair. An example of a single color cloud that displays all the colors from each frame of a video is shown. 1 shows an example of a color cloud with color and frequency information. We show examples of four different color clouds created from swatches that are then overlaid onto the L*a*b* color space (projected onto the L* and b* dimensions) to visualize similarities and differences. The shared and unique colors between different hairs are shown directly in the video.

This disclosure describes a physical device and proprietary algorithms that can quantify the optical reflectance of any material, such as human head hair swatches and real-life hair, in a manner that is highly reproducible, portable, and can operate in any lighting environment. The device, called the Cradle, can be used on a material (such as hair) and the material is analyzed with a color cloud algorithm to obtain an overall digital representation of the material, defined herein as a color cloud. This color cloud can be compared to other reflective materials at different times, or other color clouds of the same reflective material (e.g., colored hair before and after washing). The embodiments disclosed herein extract the existing colors from a video capture of the material (and do not manipulate them in any arithmetic way to produce a net result). The technology can use a robust set of colors as a basis for detecting subtle color changes.

Figure 1 shows a use case in one embodiment. A user 101 takes a cradle 102 and drags it over a hair 103. As the cradle 102 is dragged over the user's hair 103, a video of the hair 103 can be captured. This video is analyzed using a color cloud algorithm to generate a color cloud.

Figure 2 is a flow chart through one embodiment. At S202, a cradle is used to capture a video of a material such as a user's hair. For example, to create a color cloud for a user's hair, the user can move the cradle along the entire length of the hair (i.e., from the root to the tip of the hair). At S204, a color cloud is created from one or more frames of the video. Essentially, all the colors that were in one or more frames are extracted and displayed in the color cloud. At S206, the color cloud is compared to other color clouds. This comparison can be used for color matching and/or color differentiation.

A cradle is a device that allows for the capture of a video of a material so that a color cloud of the material can be created. An example of a cradle is shown in FIG. 3 (solid lines represent the exterior of the cradle, dashed lines represent the interior). The cradle has a first end 301, a second end 302, a window 303 on the second end 302, and a wall 304 connecting the first end 301 to the second end 302. In one embodiment, the first end 301 and the second end 302 are at opposite ends of the cradle. An image sensor, such as a video recording device 305, can be attached to the interior of the cradle at the first end 301 to capture video of any material 306 (e.g., hair) that is visible through the window 303. Additionally, the cradle can have a processing circuit 308 connected to the video recording device 305 and configured to create a color cloud from the video of the material 306 and compare the created color cloud to a previous color cloud. The second end 302 of the cradle can be colored with a set of predetermined colors 309 adjacent the window 303 for color separation when captured by the video recording device 305. The video can capture both the material 306 through the window 303 and the predetermined colors 309 adjacent the window 303, and then the predetermined (i.e., known) colors can be used for reference when determining the color of the material 306. A light 307 can also be attached to the cradle to provide a sufficiently bright view from the video recording device 305 to the window 303. The cradle can be used to maintain a consistent distance between the video recording device 305 and the material 306 when taking the video. The cradle can also have a hand grip 310.

The processing circuitry 308 may include a color cloud unit and an important characteristic unit. The color cloud unit may include circuitry coupled to the image sensor and configured to generate a three-dimensional virtual representation of color information associated with a plurality of digital images of an object, such as regions indicative of color frequency, color variation, or color intensity distribution associated with the object. The important characteristic unit may include circuitry for identifying one or more important characteristics of the object and generating one or more virtual instances on a graphical user interface display indicating the identity of the important characteristics of the object based on one or more inputs associated with the three-dimensional virtual representation of the color information of the object. In one embodiment, the object may be hair, and the three-dimensional virtual representation may be hair color information and/or hair characteristic information associated with a plurality of digital images of hair. Additionally, the three-dimensional virtual representation may include voxels of various intensities to indicate the color frequency of a particular characteristic associated with the imaged object. Examples of characteristic information include object identification data, object characteristic data, color frequency data, color variation data, color intensity data, presence or absence of color data, frame-by-frame or pixel-by-pixel color analysis information, and the like. Non-limiting examples of color information may include hue, tint, tone, shade, and the like. In one embodiment, a hue of a base color is a lighter version of that color, and a shade is a darker version. In one embodiment, a tone refers to the brightness (hue) or darkness (shade) of a base color. If the image object is hair, the three-dimensional virtual representation can include hair characteristic information including voxels indicating the extent of gray hair, luster, shine, and uniformity associated with multiple digital images of hair. Examples of hair characteristics include hair damage state, artificial colorant concentration, color distribution, extent of gray hair, texture, luster, distribution density, scalp characteristics, etc.

The body of the cradle in FIG. 3 was essentially a hollow rectangular prism with a window 303 at one end. In one embodiment, when window 303 is covered, the only light in the body of the cradle may be from light 307. This allows for consistent lighting conditions within the cradle. Additionally, in other embodiments, the cradle may have different shapes such as a cylinder, a cube, a trapezoidal prism, etc. The cradle may also have a screen that can be used to visualize the color cloud.

In one embodiment, the first end of the cradle may be configured to use the camera of a mobile device (e.g., a smartphone) as a video recording device. An example is shown in FIG. 4, where the first end 401 of the cradle may be an attached smartphone 402. A camera 403 on the smartphone 402 may capture video of the material 404 through a window 405 on the second end 406. Additionally, a color cloud may be created using processing circuitry already present in the smartphone 402. The color cloud and other relevant information may be visualized directly on the smartphone screen. Additionally, the smartphone flashlight 407 may be used to provide a sufficiently bright view from the smartphone camera 403 to the window 405. The cradle may also have a holder 408 that allows the smartphone 402 to be properly attached to the cradle during use and removed from the cradle when not in use.

The video captured can be of the material for which it is desired to generate a color cloud. The cradle should be held so that the video recording device can capture video of the material through the opening in the cradle provided by the window. The window can be held directly against the material so that it can block outside light from entering the interior of the cradle. An example of a material can be hair on the head. In that case, the cradle can start by capturing video of the roots and then dragged over the hair to the tips.

In one embodiment, the video containing multiple frames may be at least 6 seconds long and at least 60 frames per second. If video of human hair or a hair swatch is captured, the hair may be straight or curly.

Capturing a video of material includes capturing a video of someone's hair. Examples are shown in FIGS. 5A and 5B. Referring first to FIG. 5A, video of the entire length of a model's hair is captured, starting from the root 51a of the hair and proceeding downward to the tip 52a of the hair. FIG. 5B shows each frame of the captured video (from FIG. 5A) laid out in chronological order, with the top left frame 51b being the root 51a of the hair and the bottom right frame 52b being the tip 52a of the hair. In another embodiment, capturing a video of material may include capturing video of less than the entire length of someone's hair (e.g., capturing only half of someone's hair). Note that the video may be captured in a direction other than downward, such as proceeding upward or at an angle.

A color cloud algorithm may be used to create a color cloud from one or more frames of a video. The color cloud algorithm may extract color information from one or more frames. The color cloud includes all colors present in one or more of the multiple frames. In another embodiment, the color cloud includes all colors present in one or more of the multiple frames and all frequencies of colors present in one or more of the multiple frames. In another embodiment, the color cloud includes all colors present in one or more of the multiple frames and all frequencies of colors present in one or more of the multiple frames, where the frequencies exceed a minimum predetermined threshold.

In one embodiment, the color cloud algorithm can generate color clouds by extracting all colors from a single frame of video. For example, in FIG. 5B, a separate color cloud can be created for each frame. The color cloud created for each frame can include all colors present in that single frame, and in another embodiment, the frequencies of each color present in that single frame.

In another embodiment, the color cloud algorithm may generate a color cloud by extracting cumulative colors from a group of frames in the video (rather than from a single frame). For example, referring to FIG. 5B, a color cloud may be generated for each group of 10 frames, where the color cloud created for each group of 10 frames includes all colors present in that particular group of 10 frames, or in another embodiment, the frequency of each color present in that group of 10 frames.

Figure 6 shows an example of a color cloud, which uses a color space to display all the colors across all the frames of a video. Because the video is time-based, selecting a particular frame allows a color cloud to be created at a particular point in time and/or range of time.

Once a color cloud algorithm has extracted all the colors from one or more frames of a video, the color cloud can be displayed in a myriad of formats. For example, the colors can be displayed in a color space, which can convey color information using coordinates such as L*a*b*, RGB, LCh, HSV, etc.

In another embodiment, the color cloud can include the frequency (i.e., number of times) that each color occurs in one or more frames of the video. An example is shown in FIG. 7, where the brighter the pixel, the more frequently the color represented by that pixel and vice versa. The outer color cloud 71 (i.e., full color cloud) captures all colors that were present in one or more frames, and the inner color cloud 72 (i.e., optimized color cloud) captures colors that were present with a higher frequency. The threshold for determining whether a color is present as a higher frequency can be adjusted depending on the application. In one example, only the inner color cloud 72 is used, and low frequency (i.e., low information) colors are removed as noise.

In another embodiment, the color cloud can be used to identify hair characteristics such as gray range, gloss, shine, uniformity, lustre, etc. For example, color and frequency information from the color cloud can be used to filter out "low information" colors and enhance color detection to extract the amount of gray hair someone has. As another example, hair gloss level can be calculated by looking at the mean, variance, and spatially connected clusters of L* values in the hair color cloud, where high variability suggests a wide range of L* values from dark to light, and high spatially connected L* values suggest bands of gloss, and both pieces of information, when combined, are related to human perception of gloss.

Color clouds can be compared to other color clouds to identify similarities and differences. This allows for color change detection and color matching. An example of a comparison to other color clouds is shown in Figure 8. Color clouds are generated for two unwashed swatches and the same two swatches after they have been washed. Portions of the color clouds that overlap indicate color similarity (i.e., colors present in both swatches) and portions that do not overlap indicate color differences (i.e., colors not shared between the swatches). This technique quantifies group overlap and non-overlapping of color groups in color space (e.g., number of groups, group similarity, average group color).

Furthermore, the similarities and differences in hair color can be visualized back in the video. An example of color clouds being generated for a pre-wash and post-wash swatch is shown in Figure 9. The darker parts of the hair show common colors and the lighter parts show unique colors. The similarities and differences between these color clouds can be identified and visualized directly on the swatch. In other words, the user can play one of the videos used to capture the hair and the similarities/differences can be shown directly on the hair in the video. This is done by simply matching the color of the video pixels to colors contained in either two different color clouds or the overlap of two different color clouds. For example, the colors of the video pixels that match any color within the overlap of the color clouds or the intersection of the color clouds are colored darker in the visualization video.

In another embodiment, the atlas of hair swatches can have a directory of color clouds created for each hair swatch. Then, any subsequent capture of a hair will generate a color cloud that is compared to the swatches in the atlas to determine the closest match (i.e., most similar color). For good discrimination, a swatch should resemble itself more than other swatches.

The technologies referred to herein can capture a wide range of reflectances (e.g., from hair swatches and heads of hair), accurately and robustly measure heterogeneous materials such as color, and accurately and robustly measure hair color change. Applications of these technologies can include measuring color change more accurately and reproducibly than state-of-the-art instruments (e.g., spectrophotometers), measuring human hair color (as hair colorists do), measuring the extent of gray hair (e.g., measuring the amount of gray hair at someone's roots to indicate how much hair color product to use), measuring hair health (e.g., based on gloss level, where too little gloss indicates dullness and suggests damaged hair), dynamic visualization of hair before and after coloring and/or washing, and more. The technologies referred to herein can also enable home consumer color assessment, in-store display and visualization, and home consumer product recommendations. For example, consumers can download an app to their mobile phone to capture video and create a color cloud. The app can further recommend hair products based on certain characteristics of the hair (e.g., recommending a particular hair color product based on the user's hair color, gray level, and research performed in the app). Additionally, the cradle can have the ability to be portable (e.g., a foldable cardboard cradle, a 3D printed cradle, etc.).

Claims (6)

1. A system for creating a color cloud for a user's hair, comprising:
a cradle having a first end, a second end having a window, and at least one wall connecting the first end to the second end;
an image sensor attached to the first end of the cradle, the image sensor configured to capture a video of the user's hair as seen through the window, the video including a plurality of frames captured by the image sensor as the cradle moves across the user's hair;
a processing circuit operably coupled to the image sensor, the processing circuit comprising:
generating a three-dimensional virtual representation of color information associated with the plurality of frames of the video, the three-dimensional virtual representation including all colors present in the plurality of frames of the video and a frequency with which each color is present in the plurality of frames; and configured to identify one or more significant characteristics of the hair based on one or more inputs associated with the three-dimensional virtual representation of the hair color information and generate one or more virtual instances on a graphical user interface display that indicate the identity of the significant characteristics of the hair ;
Colors present in the plurality of frames with a frequency below a predetermined threshold are removed as noise from the three-dimensional virtual representation . The system of claim 1, wherein the image sensor is a camera on a mobile device mounted to the first end of the cradle. The system of claim 1, further comprising a light source attached to at least one of the first end, the second end, and the at least one wall. The system of claim 3, wherein the light source is a flashlight on a mobile device attached to the first end of the cradle. The system of claim 1, further comprising a holder attached to the first end of the cradle for holding the image sensor. The system of claim 1, wherein the second end is colored with multiple colors adjacent the window for color separation.

Images