JP6545658B2 - Estimating bilirubin levels - Google Patents

Description

(Related application)
This application claims the benefit of US Provisional Application No. 61 / 777,097, filed March 12, 2013, which is incorporated herein by reference.

  An estimated 60-84% of neonates develop neonatal jaundice resulting in yellowing of the skin caused by accumulation of excess bilirubin (hyperbilirubinemia), a naturally occurring compound produced by the destruction of red blood cells. The condition is typically harmless and resolves within a few days, but highly elevated bilirubin levels are devastating characterized by deafness, cerebral palsy, severe developmental delay, or even death. And irreversible neurological symptoms, which can lead to kernicterus.

  Current approaches to monitoring bilirubin levels in infants typically require repeat testing in a hospital setting. Blood concentration of bilirubin is transepithelial bilirubin measurement (TcB) measurement performed with serum total bilirubin (TSB) measured from blood samples or using a non-invasive but costly instrument Can be determined. These tests are often unavailable in resource-poor environments and thus prevent early detection and treatment of kernicterus. Visual assessment, frequently used as a substitute for these tests, is often inaccurate and can be confused by factors such as lighting or skin tone. Therefore, an improved approach is needed to provide non-invasive and cost-effective screening for excess bilirubin levels.

  Systems, methods, and devices for estimating bilirubin levels are provided. In many embodiments, a mobile device is used to capture image data of a patient's skin and a color calibration target. Image data is processed to generate an estimate of bilirubin levels. Image processing transforms the image data into multiple different color spaces and facilitates assessment of the overall yellowness of the skin while compensating for color differences caused by lighting, skin tone, and other potentially confounding factors Can be included. The screening techniques described herein are practiced by users (eg, parents, healthcare workers, community health employees) in an outpatient setting (eg, the patient's home) without requiring specialized medical equipment. Thus, the convenience, convenience, and cost-effectiveness of bilirubin monitoring can be improved.

  Thus, in a first aspect, a method is provided for estimating the level of bilirubin in a patient. The method includes receiving image data for at least one image that includes an area of the patient's skin and a color calibration target. Color balanced image data for the skin area is generated based on the color calibration target and a subset of the image data corresponding to the skin area. Bilirubin levels in the patient are estimated based on color balanced image data for the skin area. In many embodiments, image data can be obtained using any suitable imaging device. The imaging device may collect image data independently from additional attachments or equipment, such as external lenses, filters, or other specialized hardware.

  Bilirubin levels can be estimated using only image data of the patient's skin color at a particular time point, or by comparing skin color image data with baseline skin color image data. For example, the method further receives baseline skin color data (eg, about zero when baseline data is obtained within 24 hours after birth) for the corresponding patient when the patient has a reference bilirubin level It can include steps. Bilirubin levels can be estimated based on one or more differences between baseline skin color data and color balanced image data for the skin area. Baseline skin color data for a patient can be generated by capturing baseline image data for the patient when the patient has a reference bilirubin level. The baseline image data can correspond to at least one image, including a skin area and a baseline color calibration target. Color balancing baseline image data for a skin area can be generated based on the baseline color calibration target and a subset of baseline image data corresponding to the skin area. Baseline skin color data can be generated based on color balanced baseline image data for the skin area.

  Standardized color calibration targets can be used to facilitate the color balancing process. The color calibration target can include a plurality of standardized color areas, including white areas. The standardized color areas can include black areas, gray areas, light brown areas, cyan areas, magenta areas, yellow areas, and dark brown areas. The color calibration target may at least partially define an opening configured to expose the skin area and allow capture of image data for the skin area. The standardized color areas can be arranged in a known arrangement surrounding the openings. Thus, the process for generating color balanced image data for the skin area processes the received image data, and the subset of image data corresponding to the exposed skin area and the subset of image data corresponding to the white area A step of identifying can be included. The white area data can be processed to determine the observed color values for the white area. Color balanced image data for the exposed skin area can be generated based on the observed color values for the white area. The observed color values for the white area may comprise any suitable color space values, such as red, green, blue (RGB) color space values.

  Image data can be converted to multiple different color spaces to detect skin yellowing. For example, color balanced image data for the skin area can include RGB color space data, and the method of estimating the level of bilirubin in the patient further converts the RGB color space data to at least one other color space, The method may include the step of generating color balancing image data for the exposed skin area for at least one other color space. The at least one color space can include (a) cyan, magenta, yellow, and black (CMYK) color spaces, (b) YCbCr color spaces, and / or (c) Lab color spaces.

  A plurality of chromatic and achromatic features can be generated based on the image data. In some cases, the received image data can include images obtained using flash illumination and images obtained without flash illumination. Estimating the bilirubin level includes processing the plurality of normalized chromatic and achromatic features and selecting a first estimated range of the bilirubin level from one of a plurality of different bilirubin ranges Can. The features may be processed using an approach based on a selected first estimation range of bilirubin levels to generate a final estimate of bilirubin levels. The plurality of different bilirubin ranges can include low range, mid range, and high range. The plurality of features can include selected color values of the skin area for a plurality of different color spaces. In some cases, the plurality of features can include the calculation of color gradients across the skin area.

  Estimating bilirubin levels can involve performing one or more regressions. For example, the steps of processing the features and selecting the first estimated range of bilirubin levels include (a) linear regression, (b) encapsulated k nearest neighbor regression, (c) lasso regression, (d) LARS regression, e) at least one of elastic net regression, (f) support vector regression using a linear kernel, (g) support vector regression that assigns higher weights to higher bilirubin values, and (h) random forest regression Can include the steps of performing a series of regressions. The steps of processing the features and selecting a first estimated range of bilirubin levels are: (a) linear regression, (b) encapsulated k nearest neighbor regression, (c) lasso regression, (d) LARS regression, (e) Perform a series of regressions, including elastic net regression, (f) support vector regression using a linear kernel, (g) support vector regression that assigns higher weights to higher bilirubin values, and (h) random forest regression It can include steps. Using the processing approach based on the selected first estimation range of the bilirubin level selects a plurality of normalized chromatic and achromatic color features and the bilirubin level as features for final random forest regression The step of performing a final random forest regression may be included, using the first estimated range. In some cases, estimating the bilirubin level comprises determining a color space value for the skin of the patient and estimating the bilirubin level using a processing approach based on the determined color space value. The regression equation can also include features from the baseline image (eg, color space values from one or more color spaces).

In another aspect, a mobile device is provided that is configured to estimate the level of bilirubin in a patient. The device includes a camera operable to capture image data for a field of view, a processor operatively coupled to the camera, and a data storage device operatively coupled to the processor. The data storage device can store instructions that, when executed by the processor, cause the processor to receive image data captured by the camera, the image including an area of the patient's skin and a color calibration target. The instructions cause the processor to generate color balanced image data for the skin area based on the color calibration target and the subset of image data corresponding to the skin area, and bilirubin in the patient based on the color balanced image data for the skin area The level can be estimated. In many embodiments, the mobile device is used to estimate bilirubin levels independently of the mobile device (eg, lens, filter) or any external attachment to any other specialized mobile device device Can.

  The color calibration target can define an opening that is configured to at least partially expose the skin area and allow capture of image data for the skin area, and includes a plurality of standardizations, including white areas Can include a colored area. The instructions may cause the processor to process the received image data and identify a subset of image data corresponding to the exposed skin area and a subset of image data corresponding to the white area. The white area data can be processed to determine the observed color values for the white area. Color balanced RGB image data can be generated for the exposed skin area based on the observed color values for the white area. Color balanced image data for exposed skin areas can be generated for at least one other color space by converting the color balanced RGB image data to at least one other color space. The plurality of normalized chromatic and achromatic features may be processed to select a first estimated range of bilirubin levels from one of a plurality of different bilirubin ranges. The features may be processed using an approach based on a selected first estimation range of bilirubin levels to generate a final estimate of bilirubin levels.

  Alternatively, or in addition, color balanced image data for the exposed skin area can be processed to determine color space values for the patient's skin. Multiple normalized chromatic and achromatic features can be processed using an approach based on the determined patient's skin color space values to estimate bilirubin levels.

  The mobile devices described herein can further include a flash unit operable to selectively illuminate the field of view. The received image data processed to estimate bilirubin levels may include an image captured using the field of view illuminated by the flash unit and an image captured using the field of view not illuminated by the flash unit it can.

  In another aspect, a method is provided for estimating bilirubin levels in a patient. The method includes receiving, from the mobile device, image data regarding the image, including the skin area of the patient and the color calibration target. Color balanced image data for the skin area may be generated based on the color calibration target and a subset of the image data corresponding to the skin area via one or more processors. Bilirubin levels in a patient can be estimated based on color balanced image data for the skin area via one or more processors. The estimated bilirubin level can be transmitted to the mobile device. In many embodiments, at least one of the steps of receiving image data and transmitting the estimated bilirubin level is performed using short message service (SMS) text messaging. Image data can be obtained by the mobile device without the use of external attachments, hardware add-ons, or any other specialized equipment.

Other objects and features of the present invention will become apparent upon review of the specification, claims and attached figures.
This specification provides, for example, the following.
(Item 1)
A method of estimating the level of bilirubin in a patient, comprising
Receiving image data for at least one image, including an area of skin of the patient and a color calibration target;
Generating color balanced image data for the skin area based on the color calibration target and a subset of the image data corresponding to the skin area;
Estimating bilirubin levels in the patient based on color balanced image data for the skin area
Method, including.
(Item 2)
The method further includes receiving baseline skin color data for the patient corresponding to the patient having a reference bilirubin level, wherein estimating the bilirubin level comprises: calculating the baseline skin color data and the color for the skin area The method according to item 1, wherein the one or more differences between the balanced image data are based.
(Item 3)
Baseline skin color data for the above patients is
Capturing baseline image data for the patient when the patient has the reference bilirubin level, wherein the baseline image data comprises at least one image including the skin area and a baseline color calibration target Corresponding step and
Generating color balancing baseline image data for the skin area based on the baseline color calibration target and a subset of the baseline image data corresponding to the skin area;
Generating the baseline skin color data based on the color balanced baseline image data for the skin area;
The method according to item 2, generated by
(Item 4)
The method of claim 1, wherein the color calibration target comprises a plurality of standardized color areas, including white areas.
(Item 5)
5. A method according to item 4, wherein the standardized color area comprises black area, gray area, light brown area, cyan area, magenta area, yellow area and dark brown area.
(Item 6)
5. A method according to item 4, wherein the color calibration target is at least partially configured to expose the skin area and to enable capture of image data for the skin area.
(Item 7)
7. A method according to item 6, wherein the standardized color areas are arranged in a known arrangement surrounding the opening.
(Item 8)
Generating color-balanced image data for the skin area;
Processing the received image data to identify a subset of image data corresponding to the exposed skin area and a subset of image data corresponding to the white area;
Processing the white area data to determine observed color values for the white area;
Generating color balanced image data for the exposed skin area based on the observed color values for the white area.
6. A method according to item 6, including
(Item 9)
9. The method of item 8, wherein the observed color values for the white area comprise red, green, blue (RGB) color space values.
(Item 10)
The color balanced image data for the skin area includes RGB color space data, and the method further converts the RGB color space data to at least one other color space, for the at least one other color space A method according to claim 1, comprising the step of generating color balanced image data for the exposed skin area of
(Item 11)
Item 10, wherein the at least one other color space comprises (a) cyan, magenta, yellow, and black (CMYK) color spaces, (b) YCbCr color spaces, or (c) Lab color spaces the method of.
(Item 12)
11. The method of item 10, wherein the at least one other color space comprises (a) cyan, magenta, yellow (CMY) color space, (b) YCbCr color space, or (c) Lab color space.
(Item 13)
12. A method according to item 11, wherein the received image data comprises an image obtained using flash illumination and an image obtained without flash illumination.
(Item 14)
The step of estimating the bilirubin level is
Processing a plurality of normalized chromatic and achromatic features and selecting a first estimated range of the bilirubin level from one of a plurality of different bilirubin ranges;
Processing the features using an approach based on the selected first estimation range of the bilirubin level to generate a final estimate of the bilirubin level;
14. A method according to item 13, including
(Item 15)
15. The method of item 14, wherein the plurality of different bilirubin ranges include low range, mid range, and high range.
(Item 16)
15. The method of item 14, wherein the plurality of features include selected color values of the skin area for a plurality of different color spaces.
(Item 17)
15. The method of item 14, wherein the plurality of features include calculation of color gradients across the skin area.
(Item 18)
The steps of processing the features and selecting the first estimated range of bilirubin levels are: (a) linear regression, (b) encapsulated k nearest neighbor regression, (c) lasso regression, (d) LARS regression, e) at least one of elastic net regression, (f) support vector regression using a linear kernel, (g) support vector regression that assigns higher weights to higher bilirubin values, and (h) random forest regression 15. A method according to item 14, comprising performing a series of regressions.
(Item 19)
The steps of processing the features and selecting the first estimated range of bilirubin levels are: (a) linear regression, encapsulated k nearest neighbor regression, (c) lasso regression, (d) LARS regression, (e) elasticity Step through a series of regressions, including net regression, (f) support vector regression using linear kernels, (g) support vector regression that assigns higher weights to higher bilirubin values, and (h) random forest regression 15. A method according to item 14, including
(Item 20)
The step of using the processing approach based on the selected first estimation range of the bilirubin level comprises, as features for the final random forest regression, the plurality of normalized chromatic and achromatic features and the bilirubin level. 15. A method according to item 14, comprising the step of performing a final random forest regression using the selected first estimation range of.
(Item 21)
The step of estimating the bilirubin level may include determining a color space value for the skin of the patient and estimating the bilirubin level using a processing approach based on the color space value of the skin of the patient determined above. , Method according to item 1.
(Item 22)
A mobile device configured to estimate the level of bilirubin in a patient,
A camera operable to capture image data for a field of view;
A processor operatively coupled to the camera;
A data storage device operatively coupled to the processor, wherein the data storage device, when executed by the processor,
Receiving image data for an image captured by the camera, the image including an area of the patient's skin and a color calibration target;
Generating color balanced image data for the skin area based on the color calibration target and a subset of the image data corresponding to the skin area;
Estimating bilirubin levels in the patient based on color balanced image data for the skin area
A data storage device storing instructions for causing
A device comprising:
(Item 23)
The color calibration target defines an opening, including a white area, configured to at least partially expose the skin area and enable capturing image data for the skin area. And the instruction is for the processor to:
Processing the received image data to identify a subset of image data corresponding to the exposed skin area and a subset of image data corresponding to the white area;
Processing the white area data to determine observed color values for the white area;
Generating color balanced RGB image data for the exposed skin area based on the observed color values for the white area;
Generating color balanced image data for the exposed skin area for the at least one other color space by converting the color balancing RGB image data to the at least one other color space;
Processing a plurality of normalized chromatic and achromatic features and selecting a first estimated range of the bilirubin level from one of a plurality of different bilirubin ranges;
Processing the features using an approach based on the selected first estimation range of the bilirubin level to generate a final estimate of the bilirubin level
Item 22. The mobile device according to item 22.
(Item 24)
The color calibration target defines an opening, including a white area, configured to at least partially expose the skin area and enable capturing image data for the skin area. And the instruction is for the processor to:
Processing the received image data to identify a subset of image data corresponding to the exposed skin area and a subset of image data corresponding to the white area;
Processing the white area data to determine observed color values for the white area;
Generating color balanced RGB image data for the exposed skin area based on the observed color values for the white area;
Generating color balanced image data for the exposed skin area for the at least one other color space by converting the color balancing RGB image data to the at least one other color space;
Processing the color balanced image data for the exposed skin area to determine the skin color for the patient;
Processing a plurality of normalized chromatic and achromatic features using the determined skin color based approach to estimate the bilirubin level;
Item 22. The mobile device according to item 22.
(Item 25)
The apparatus further comprises a flash unit operable to selectively illuminate the field of view, wherein received image data processed to estimate the bilirubin level is captured using the field of view illuminated by the flash unit. 24. A mobile device according to any of items 23 and 24, comprising an image and an image captured using a field of view not illuminated by the flash unit.
(Item 26)
A method for estimating bilirubin levels in a patient, comprising
Receiving, from the mobile device, image data regarding the image, including the patient's skin area and the color calibration target;
Generating color balanced image data for the skin area based on the color calibration target and a subset of the image data corresponding to the skin area via one or more processors.
Estimating bilirubin levels in the patient based on color balanced image data for the skin area via the one or more processors.
Transmitting the estimated bilirubin level to the mobile device;
Method, including.
(Item 27)
26. The method of item 25, wherein at least one of the steps of receiving the image data and transmitting the estimated bilirubin level is performed using short message service (SMS) text messaging.

The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present invention can be obtained by reference to the following detailed description that sets forth illustrative embodiments in which the principles of the invention may be utilized and the accompanying drawings. I will.
FIG. 1A illustrates guidelines for the use of phototherapy to treat neonatal jaundice according to embodiments. FIG. 1B illustrates guidelines for the use of replacement blood transfusion to treat neonatal jaundice according to an embodiment. FIG. 1C illustrates a Bhutani nomogram for assessing the risk associated with neonatal jaundice. FIG. 2 illustrates the use of a mobile device to capture image data used to estimate bilirubin levels in a patient, in accordance with many embodiments. FIGS. 3A-3D illustrate color calibration targets for use with a mobile device for capturing image data used to estimate bilirubin levels, according to many embodiments. FIGS. 3A-3D illustrate color calibration targets for use with a mobile device for capturing image data used to estimate bilirubin levels, according to many embodiments. FIGS. 3A-3D illustrate color calibration targets for use with a mobile device for capturing image data used to estimate bilirubin levels, according to many embodiments. FIGS. 3A-3D illustrate color calibration targets for use with a mobile device for capturing image data used to estimate bilirubin levels, according to many embodiments. 4A-4C illustrate an exemplary user interface of a mobile application for estimating bilirubin levels, in accordance with many embodiments. 4A-4C illustrate an exemplary user interface of a mobile application for estimating bilirubin levels, in accordance with many embodiments. 4A-4C illustrate an exemplary user interface of a mobile application for estimating bilirubin levels, in accordance with many embodiments. 5A-5F illustrate exemplary image data collected for use in estimating bilirubin levels, according to many embodiments. FIG. 6 illustrates a method for estimating bilirubin levels in a patient according to many embodiments. FIG. 7 illustrates a method for generating color balanced image data for a patient's skin, in accordance with many embodiments. FIG. 8 illustrates the identification of standardized color regions in an image according to many embodiments. FIG. 9A illustrates a method for estimating bilirubin levels in a patient, according to many embodiments. FIG. 9B illustrates another example of a method for estimating bilirubin levels in a patient, according to many embodiments. FIG. 10 illustrates a method for estimating bilirubin levels in a patient, according to many embodiments. FIG. 11A illustrates a method for estimating bilirubin levels using baseline skin color data according to many embodiments. FIG. 11B illustrates a method for generating baseline skin color data, in accordance with many embodiments. FIG. 12 illustrates a mobile device for estimating bilirubin levels in accordance with many embodiments. FIG. 13 illustrates a mobile device in communication with a data processing system for estimating bilirubin levels, in accordance with many embodiments.

  The systems, devices, and methods described herein provide an improved approach to estimating bilirubin levels in a patient (eg, an infant or an adult). Mobile devices configured with suitable software can be used to capture an image of the patient's skin and a standardized color calibration target. The skin and target image data can be used to generate color balanced image data that is analyzed to estimate bilirubin levels in the patient. For example, color balanced image data can be converted to multiple different color spaces to extract features representing skin yellowing, and these features can be converted to multiple bilirubin estimates. Can be used in the regression of The system described herein, in contrast to existing approaches for measuring bilirubin levels, which either rely on invasive blood tests or utilize costly instrumentation. Device and method allow convenient, portable, and inexpensive bilirubin level estimation that can be easily performed by non-medical personnel using personal mobile devices, thereby the convenience of bilirubin monitoring Improve gender and cost effectiveness. Advantageously, the methods described herein can be performed on the mobile device without requiring the use of external attachments, hardware add-ons, or any other specialized mobile device device. It should be noted that the disclosed techniques provide accurate bilirubin level estimates over a wide range of bilirubin concentrations, in contrast to TcB measurements, which exhibit reduced accuracy at high bilirubin levels. In addition, the methods described herein take into account skin tone as well as variability in different lighting conditions, thereby improving the accuracy and flexibility of non-invasive bilirubin level estimation. Furthermore, the use of the mobile device software platform allows easy and quick updates of the estimation methods and algorithms described herein, thus enabling refinements and upgrades to be made as needed.

  Referring now to the drawings, FIG. 1A illustrates guidelines for the use of phototherapy to treat neonatal jaundice. Similarly, FIG. 1B illustrates guidelines for the use of replacement blood transfusion to treat neonatal jaundice. Guidelines are used by healthcare professionals to determine the appropriate course of treatment based on the age of the infant, serum total bilirubin (TSB), gestational age (eg ≧ 35 weeks), and other risk factors. be able to. Risk factors may include alloimmune hemolytic disease, G6PD deficiency, asphyxia, significant sleep overactivity, temperature instability, sepsis, acidemia, or albumin levels below 3.0 g / deciliter. The curves depicted in FIGS. 1A and 1B show exemplary treatment thresholds for low risk, medium risk, and high risk infants.

  FIG. 1C illustrates a Bhutani nomogram 110 for assessing the risk associated with neonatal jaundice. Bhutani nomogram 110 can be used by health care workers to assess the risk of developing infantile hyperbilirubinemia based on the child's age at birth and bilirubin levels. For example, Bhutani nomogram 110 may include multiple percentile curves 112 used to define low risk zone 114, low intermediate risk zone 116, high intermediate risk zone 118, and high risk zone 120.

  FIG. 2 illustrates mobile device based estimation of bilirubin level in accordance with many embodiments. An imaging device such as a mobile device 202 (e.g., a smartphone, a tablet) can be used to capture an image of the skin area 204 of the patient 206 and the color calibration target 208. Preferred skin areas of the approach described herein include the forehead and sternum, as well as any other prominent flat area of skin that may be evenly illuminated. In addition, jaundice typically appears first on the forehead and progresses gradually down the body, so areas closer to the forehead are potentially more informative for diagnosis possible. Thus, in many embodiments, the color calibration target 208 is on the abdomen of the patient 206 near the sternum. The mobile device 202 has a camera (not shown) for recording image data and a user interface (UI) of a mobile software application ("mobile app" or "app") for estimating the bilirubin level based on the image data. And a display 210 used to present the The mobile device 202 does not require the user to perform the methods described herein, in addition to the color calibration target 208, other appliances or hardware in addition to the color calibration target 208, the mobile app on their personal device It can be a personal device of the user (e.g., parents, healthcare workers, local health employees, etc.), as it only needs to be installed. It should be noted that the mobile device 202 implements the method described herein independently of any further attachments or accessories to the mobile device, such as external lenses, filters, or other specialized mobile device devices. Can be used for Additional details regarding suitable software and hardware components for the mobile device 202 are provided in further detail below.

  FIGS. 3A-3D illustrate color calibration targets that can be used with a mobile device to estimate bilirubin levels according to many embodiments. The color calibration targets described herein (also known as "color calibration cards" and "color cards") may affect lighting conditions or other environmental conditions that affect the obtained color balance of skin image data. Can be used to account for differences in Referring to FIG. 3A, a color calibration target 300 can be provided on a rectangular card 302. The card 302 can be any size of card paper suitable for placement on the skin of a patient (e.g., an infant), such as approximately the size of a business card. In some cases, the card 302 can be sterilizable or disposable to prevent the spread of pathogens between patients. The color calibration target 300 can include a plurality of standardized colored areas 304 that can be printed on the card 302. Colored regions 304 can be of any suitable size, number, or shape (eg, square, rectangular, polygonal, circular, oval, etc.). For example, color calibration target 300 is depicted to include eight similarly sized square colored areas 304a-h. The standardized colored areas 304 may each be of a different color (eg, black, gray, white, cyan, magenta, yellow, light brown, dark brown) and located in a known arrangement on the card 302 it can. For example, in the embodiment of FIG. 3A, 304a is a black area, 304b is a gray area (eg, 50% gray), 304c is a white area, and 304d is a light brown area ( For example, the first skin tone) 304e is a cyan area 304f is a magenta area 304g is a yellow area 304h is a dark brown area (e.g. Second skin tone). Other color arrangements and combinations can also be used. Additionally, the back side (not shown) of the card 302 can include one or more adhesive areas that allow the card 302 to be removably attached to the patient's skin. The back side can also include related instructions such as instructions for the user regarding how to download the accompanying mobile app on the personal mobile device.

  FIG. 3B illustrates an alternative embodiment of a color calibration target 320. The color calibration target 320 is substantially similar to the color calibration target 300 but also defines an opening 322. The opening 322 can be positioned such that the skin area of interest is exposed through the opening 322 when the color calibration target 300 is deployed on the patient. The openings 322 can be fully defined (eg, completely enclosed) or partially defined (eg, partially enclosed) by the target 320. A plurality of standardized color areas 324, including white areas, can be positioned around the opening 322 in a known arrangement. The embodiment of FIG. 3B further includes light gray, mid gray and dark gray areas for a total of 10 different standardized color areas. FIG. 3C illustrates the color calibration target 340 provided on the square card 342, with a plurality of standardized color areas 344 surrounding the square opening 346. Some of the color regions 344 can include two or more colors, such as color region 348, including a central black square bounded by white. FIG. 3D illustrates a color calibration target 360 having a plurality of standardized color areas 362 surrounding an opening 364. The openings 364 can be adjacent to part or all of the color area 362. Color area 362 is depicted in FIG. 3D as a series of rectangular areas with different aspect ratios. The color, geometry, and arrangement of the color regions of the color calibration target described herein can be selected based on the image processing and analysis method to be used, such as the embodiments discussed below. .

  4A-4C illustrate an exemplary user interface (UI) of a mobile app for estimating bilirubin levels, in accordance with many embodiments. FIG. 4A illustrates an overview UI 400 for displaying various statistics and metrics associated with a patient. UI 400 may include patient identification information 402 (eg, patient name, study ID number), birth time 404, and any available bilirubin level results 406 (eg, TSB and / or TcB results). The UI 400 may also include a button 408 or other interactive element that allows the user to collect patient image data (eg, video samples or photo samples). In some cases, video samples may be advantageous to eliminate motion blur problems. FIG. 4B illustrates an instruction UI 420 to assist the user in capturing patient image data. The UI 420 can include graphical and / or textual instructions 422 that direct the user to perform the appropriate steps. For example, the instructions 422 can instruct the user to place a color calibration target on the patient's abdomen below the sternum. In many embodiments, the instructions 422 are adapted to be easily understood by the individual without medical training, thereby enabling non-medical workers to operate the mobile app. FIG. 4C illustrates a live preview UI 440 displayed to the user during the image capture process. The UI 440 can include a video preview 442 of the current view of the mobile device's camera. The UI 440 may include one or more positioning targets 444 to assist the user in positioning the mobile device. For example, positioning target 444 can be a box, frame or colored area overlaid on video preview 422 to indicate proper placement of a color calibration target. The size and location of positioning target 444 is selected to constrain the distance of the mobile device from the patient within the optimal range and to ensure that the field of view properly captures the appropriate skin area and color calibration target. Can be Once the field of view is properly aligned, the user can record image data by tapping the record button 446. Additional details regarding the image acquisition and analysis process are provided below.

  5A-5F illustrate exemplary image data collected by a mobile app according to many embodiments. Following the recording process, the mobile app can display image data to the user for review and approval. Furthermore, to ensure that the captured image is of sufficient quality for processing and analysis, the mobile app includes functionality to detect image quality and the user for images that do not meet the quality threshold Alert, which can force re-shooting. FIG. 5A illustrates an image 500 of sufficient quality in which the illumination is satisfactory and both the patient and the color calibration target are clearly visible. FIG. 5B illustrates an insufficient image 502 in which a portion of the image is disturbed by glare. FIG. 5C illustrates an insufficient image 504 where the image is too bright. FIG. 5D illustrates an insufficient image 506 in which the color calibration target is partially disturbed. FIG. 5E illustrates an insufficient image 508 where the image is partially obstructed by shadows. FIG. 5F illustrates an insufficient image 510 where the image is too dark.

  FIG. 6 illustrates a method 600 for estimating bilirubin levels in a patient, in accordance with many embodiments. The method 600, as well as all other methods described herein, includes the mobile device or a separate computing system (eg, remote server) in communication with the mobile device, such as the systems and devices disclosed herein. It can be practiced using either. Certain steps of method 600, as well as all other methods described herein, may be optional, or combined with suitable steps of other methods disclosed herein, or May be substituted for

  At act 610, image data is received for at least one image that includes an area of the patient's skin and a color calibration target. Image data may be collected using the mobile device's camera, as described hereinabove. The image data can include photographic data, video data, or any suitable combination thereof. Photo data and video data can be captured sequentially or simultaneously (eg, images are taken during video recording). Image data may be obtained using and / or without flash illumination. In some cases, flash illumination can be used to cancel ambient illumination so that the illumination in the obtained image is only determined by the contribution from the flash illumination. This may be advantageous in order to produce more consistent illumination, or in situations where ambient illumination is suboptimal (eg too dark, noticeable colored etc).

  In many embodiments, image data is collected using a mobile app that controls the flash unit of the mobile device and the sequence and number of images captured. For example, the app allows the user to assess the amount of glare in the image (eg, via a video preview) during initial positioning of the mobile device, as needed, to reduce or eliminate glare. The flash unit can be turned on so that it can be repositioned. During the recording process, the app controls the mobile device and first turns on the flash unit to obtain image data and then turns off the flash unit to obtain image data, thereby respectively emitting the flash Two sets of images can be generated, with and without. For example, when the user starts the image capture process (e.g. by pressing the record button), the app will turn on the flash unit in the first half and off in the second half to video the patient and the calibration target It can be configured to record. The app can also capture two photos, taken during the first and second half of the video recording, respectively. The overall length of the video can be any suitable time, such as about 10 seconds. The timing of the image capture process can further be configured to ensure that the mobile device's image sensor (eg, charge coupled device (CCD) array) stabilizes before the next image is taken . For example, the app may include a defined delay time (e.g., 3 to 4 seconds) before recording each image set.

  As mentioned above, once the image capture sequence is complete, the mobile app can analyze the image data to determine if it is of sufficient quality for subsequent use. For example, the app may implement suitable image analysis techniques (eg, computer vision) to assess image quality. An exemplary procedure extracts a color calibration target from the captured image data and checks color consistency across each color area of the calibration target (e.g., standard deviation of pixel values for each color area is below a predetermined threshold) Determine if the user is not recommended to take any image that does not pass the exam. In some cases, the app can be configured to capture multiple sets of image data per session to maximize the likelihood that at least some of the image data will meet the quality criteria. The approved image may then be processed on the mobile device or transmitted to a separate computing system for processing, as described in further detail below.

  At act 620, color balanced image data for the skin area is generated based on the color calibration target and a subset of the image data corresponding to the skin area. Color balancing is performed to compensate for different lighting conditions as the color content of the image data can vary significantly based on ambient lighting (eg intensity, color, type (halogen fluorescence, natural etc)) be able to. In many embodiments, the image data is first normalized by dividing the three red, green, blue (RGB) color channels by the overall luminescence of the image. In addition, pixel color values observed for one or more of the standardized color areas of the color calibration target may be color balanced applied to the skin area in the image data, as discussed below. It can be used to determine the adjustment.

  At act 630, bilirubin levels in the patient are estimated based on color balanced image data for the skin area. Because hyperbilirubinemia results in yellowing of the skin, bilirubin levels can be determined based on the amount of yellow in the color-balanced skin area image data. The determination may be performed using any suitable technique, such as converting the color balanced image data into a plurality of different color spaces selected to facilitate quantification of the overall yellowness of the skin. Can. Image features generated from the transformed image data can be input into a series of machine learning regressions designed to estimate the concentration of bilirubin in the patient's body. These approaches are discussed in more detail below.

  FIG. 7 illustrates a method 700 for generating color balanced image data, in accordance with many embodiments. Method 700 can be applied to the patient's skin area and the resulting photo and / or video data of the color calibration target, as described above. At act 710, the received image data is processed to identify image data corresponding to the exposed skin area and image data corresponding to the white area. For example, the image data can be segmented to extract pixel values of the color calibration target and the color area of the skin area of interest (eg, sternum, forehead). For color calibration targets, if the mobile app UI includes a positioning target (eg, positioning target 444 in FIG. 4C) for aligning the calibration target, then the approximate pixel coordinates of the color calibration target in the image data are already known Thus, the search space for color regions can be constrained as appropriate. The positioning of the color area can be determined by identifying at least two color areas on the calibration target.

  FIG. 8 illustrates the identification of standardized color regions in an image according to many embodiments. The color domain segmentation process can be implemented via a suitable algorithm configured to apply a predetermined threshold to the image. In many embodiments, the cyan, magenta, and yellow regions have distinct hues and relatively high saturation, so the algorithm first attempts to identify at least two of these regions. The algorithm transforms the image data into a hue saturation value (HSV) color space and applies empirically determined thresholds to the hue and saturation channels, thereby obtaining a threshold hue image 800 and a threshold saturation image 802. it can. An "AND" operation can be performed on the two threshold images to separate a defined color area from the rest of the image, thereby producing a segmented image 804. Furthermore, since the approximate size of each color area is determined in advance, this information can be used to distinguish the color area from noise in the image data (e.g. using edge detection, morphological operations etc.) do it). The algorithm can then use contour detection to identify the boundaries of the color region, and contour smoothing is performed using a suitable technique such as the Douglas-Peuker algorithm. Because the overall arrangement of color areas is known, once the positions of the two color areas are determined, the general orientation of the calibration target is to extrapolate the positions of other areas, including white areas. It can be calculated and used.

式中、Ｋ_Ｗ＝（Ｒ’_Ｗ，Ｇ’_Ｗ，Ｂ’_Ｗ）は、色較正標的に関する白色領域の未加工観察色値である。 At act 720, the white area data is processed to determine the observed color values for the white area. At act 730, color balanced image data for the exposed skin area is generated based on the observed color values for the white area. The observed color values for the white area can include RGB color values, which can be used to adjust the RGB values of the image for the skin area. For example, in the case of the color values x = (R ′, G ′, B ′) observed for the white area, the color values (R, G, B) adjusted for the skin area can be obtained by:

Where K _W = (R ′ _W , G ′ _W , B ′ _W ) is the raw observed color value of the white area for the color calibration target.

  FIG. 9A illustrates a method 900 for estimating bilirubin levels in a patient, in accordance with many embodiments. Method 900 can be practiced using color balanced RGB image data obtained as described herein above. In act 910, the color balanced RGB image data is converted to at least one other color space to produce color balanced image data for the exposed skin area for the at least one other color space. As mentioned above, the yellowing of color-balanced skin area image data correlates with bilirubin levels in the patient's body. In many embodiments, an image sensor (eg, a CCD or CMOS sensor) of the mobile device interpolates the reflected light into red, green and blue wavelengths, and the image sensor clearly captures reflections in the yellow wavelength band Can be prevented. Therefore, color-balanced RGB data is converted to multiple different color spaces, such as cyan, magenta, and yellow (CMY); cyan, magenta, yellow, and black (CMYK); YCbCr; or Lab color space It can be done. Any suitable number or combination of color spaces can be used. For example, in many embodiments, RGB image data is converted to CMYK, YCbCr, and Lab color spaces. Alternatively, RGB image data can be converted to CMY, YCbCr, and Lab color spaces.

  At act 920, a plurality of normalized chromatic and achromatic features are processed to select a first estimated range of bilirubin levels from a plurality of different bilirubin ranges. The features may be chromatic and / or achromatic (e.g. luminescence) values for the skin area, each feature corresponding to a color space value of the color space used. In some cases, the features can include the calculation of one or more color gradients across the skin area. Any suitable number and combination of features can be used. For example, three features can be extracted from each of four color spaces (eg, RGB, CMY, YCbCr, Lab or RGB, CMYK, YCbCr, Lab) to obtain twelve chromatic and achromatic features. Furthermore, features can be extracted separately from the image data obtained with and without flash illumination, respectively, yielding a total of 24 chromatic and achromatic features. The features may be normalized based on the overall luminescence of the image, as described herein above with respect to operation 620 of method 600. The extracted features can be used as input to a machine learning regression used to estimate bilirubin levels. The regression can utilize some or all of the extracted features, and the optimal subset of features used is selected based on machine learning techniques. The machine learning regressions described herein can be trained on suitable data sets, such as clinical patient data, and can incorporate any suitable number and combination of parametric and non-parametric regression models. Exemplary regressions suitable for use in conjunction with the methods described herein are provided below. In many embodiments, the initially calculated feature and the output of the selected regressor classify the estimated bilirubin levels into one of a plurality of different bilirubin ranges, such as low, medium, and high range. Used for

  In act 930, the features are processed using a processing approach based on the selected first estimated range of bilirubin levels to generate a final estimate of bilirubin levels. Similar to act 920, normalized chromatic and chromatic features can be used to signal one or more machine learning regressions. The input to the regression may differ based on whether the first estimated range of bilirubin levels is low, medium or high, as provided in more detail below. For example, the classification of the first estimation range can be used as an input to a machine learning regression. The results of the initial regression performed in act 920 can also be used as input. The present "two-step" approach of bilirubin estimation can be used to generate more accurate estimation results as compared to direct estimation.

  FIG. 9B illustrates a method 950 for estimating bilirubin levels in a patient, in accordance with many embodiments. Method 950 can be practiced in combination with method 900 to obtain a more accurate estimate of bilirubin levels. Similar to method 900, method 950 can be practiced using color balanced RGB image data obtained as described hereinabove. At act 960, the color balanced RGB image data is converted to at least one other color space, as described above with respect to act 910 of method 900, for the exposed skin area for at least one other color space. Generate color balanced image data.

  At act 970, color balanced image data for the exposed skin area is processed to determine color space values for the patient's skin. The color space values for the patient's skin can be used to classify the patient's skin color into one of several different skin color types, such as light skin, neutral skin, and dark skin. The skin color type may be related to the race and / or ethnicity of the patient. The skin color type can be determined based on the color values of the skin area and / or the color calibration target obtained from the color balancing RGB image data. For example, the skin area can be compared to one or more standardized color areas (eg, first and second skin tone color areas) of the color calibration target to determine the skin color type.

  At action 980, a plurality of normalized chromatic and achromatic features are processed using the determined skin color based approach to estimate bilirubin levels. Normalized chromatic and achromatic features can be extracted from color balancing image data for one or more different color spaces, as described above for method 900. The features, as well as the action 930 of method 900, can be used as an input to a suitable machine learning regression, along with the determined skin color.

  FIG. 10 illustrates a method 1000 for estimating bilirubin levels in a patient, in accordance with many embodiments. Method 1000 includes obtaining 1002 an image from the camera, color balancing 1004 the image data, extracting 1006 features from the color balanced image data, and 1006 performing machine learning regression based on the features. , Generate bilirubin estimates 1010.

  Image data may be obtained from the camera of the mobile device under the control of the preferred mobile app (action 1002). Some of the image data can be obtained using flash illumination, and some of the image data can be obtained without flash illumination. Once the app verifies that the image is of sufficient quality for processing and analysis, the image data can be color balanced (action 1004). Color balancing may involve the identification of image data subsets corresponding to color calibration targets and the automatic segmentation of the standardized color area of the target using the threshold method described herein above (action 1012) ). The segmented white areas can be used to white balance the image data (action 1014), thereby producing color balanced image data. The color balanced image data can be RGB image data. One or more features may be extracted from the color balanced image data (action 1006). The feature extraction process may involve transforming the image data from the RGB color space into multiple different color spaces, as described hereinabove. The transformed image data can then be used to calculate a plurality of normalized chromatic and achromatic features (action 1018).

  Some or all of the extracted features can be used as input to a machine learning regression (action 1008). Any suitable number and / or combination of regressions can be used. For example, the initial set of machine learning regressions can include five different regressions (acts 1020, 1022, 1024, 1026, and 1028). For example, the first regression can include one or more encapsulated k nearest neighbor (kNN) regressions (action 1020). The regression can utilize a database of known features and bilirubin values. When unknown test vectors are analyzed, a convex hull can be found around the test vectors in the database of features. Feature points from the convex hull can then be used in linear regression. A new regression can be constructed each time a new test point is analyzed. The parameters for creating the envelope can be normalized values of luminance (e.g. from YCbCr color space conversion) and "green" channels (e.g. from RGB color space). This can be used to ensure that the convex hull calculations only occur in two dimensions and thus the number of points in the convex hull is manageable (e.g. about 4 to 6 points).

  The second regression may include one or more Lasso regressions and / or one or more minimum angular regressions (LARS) (action 1022). LARS may be useful to determine which features are most useful using a variation of forward feature selection from the total set of extracted features. For example, the best predictors from the feature set can be selected by developing a linear regression of single features from each feature. The most correlated output can be selected as the "first" feature. The prediction can be subtracted from the output to obtain a residual. The algorithm may be distinguished from other forward feature selection algorithms in that it attempts to find another feature with approximately the same correlation to the first feature to the output and the residual. This can then find the “conformal” direction between the two estimates, and a third feature that maximizes the correlation to the new residual along the conformal direction. New angles are then found from the previous features, and new features can be added to the set. Features can be added in this way until the desired accuracy is met.

  The third regression may include one or more elastic net regressions, also known as elastic net algorithms (action 1024). Elastic net regression is a combination of Lasso and ridge regression. However, instead of using only forward feature selection, the algorithm can also adopt the L1 and L2 norms for its objective function. This is associated with LARS and Lasso, but with some "back-off" regularization so that it can be more stable. Parameters can be cross-validated using a step-by-step comprehensive process.

  The fourth regression may include one or more support vector (SV) regressions (action 1026). Two SV regressions can be employed to capture possible non-linear relationships between image data and bilirubin levels. SV regression can be used to find linear regression functions in high dimensional feature space. The input data can then be mapped to space using a potential non-linear function. The first SV regression can use a linear kernel, and the second SV regression can use non-linear radial basis functions to assign higher weights to higher rates of bilirubin values.

  The fifth regression may include one or more random forest regressions (action 1028). For example, the fifth regression uses random forest regression, with 500 trees. A random forest is a set of estimators. This can use many "classification" decision trees for different subsamples of the data set. The outputs of these trees can be averaged to improve prediction accuracy and control overfitting. Each tree can be created using random subsamples (with permutations).

  In many embodiments, other types of regression may also be used in addition to, or as a substitute for, one or more of the regressions described herein. For example, one or more linear regressions can also be used. Method 1000 can be practiced using any suitable type of regression, including linear and non-linear regression.

  Following the initial regression, one or more multi-layer classifiers can be used (action 1030). For example, the initially calculated feature, together with the output of the initial set of regressors, may be used to classify the first estimated range of bilirubin levels into one of a plurality of different bilirubin ranges as described herein above. Can be used. The classifiers can include random forest classifiers, support vector machines, and k nearest neighbors (k = 3). The results of all classifiers as well as the log likelihood per class can be used as a feature for final stack regression, which may be final random forest regression (action 1032). The originally extracted features and results of the initial regression can also be used as features for this final stage regression. The final regressor can be trained using a suitable machine learning algorithm such as AdaBoost. The final regressor can be used as a final estimate of bilirubin level 1010 (eg, measured in milligrams / deciliter). In order to avoid overfitting, single-pass cross validation can be used at all learning levels.

  FIG. 11A illustrates a method 1100 for estimating bilirubin levels in a patient using baseline skin color data, in accordance with many embodiments. Method 1100 is similar to method 600, but the patient's current skin color data is compared to baseline skin color data to generate a bilirubin estimate. This approach can be used to compensate for factors that may confuse image analysis, such as different skin tones depending on the race or ethnicity of the patient.

  At act 1110, baseline skin color data for the patient is received, which corresponds to when the patient has a reference bilirubin level. Reference bilirubin levels can be known bilirubin levels (eg, based on TSB or TcB tests). If the patient is an infant, baseline skin color data can be collected within the first 24 hours of the infant's birth, when bilirubin levels are very low, typically around zero. Methods suitable for collecting and generating baseline skin color data are described below.

  In act 1120, image data is received for at least one image that includes an area of the patient's skin and a color calibration target. In act 1130, color balanced image data for the skin area is generated based on the color calibration target and a subset of the image data corresponding to the skin area. The image data acquisition and color balancing processes may be similar to those described herein above with respect to acts 610 and 620 of method 600, respectively. Actions 1120 and 1130 can be performed at any time after baseline skin color data is received, and generate a series of image data sets used to determine if the skin is more yellow. As such, it can be repeated for any suitable period of time (eg, the first 4-5 days from the birth of the infant).

  At act 1140, bilirubin levels in the patient are estimated based on the difference between the baseline skin color data and the color balanced image data for the skin area. In many embodiments, baseline skin color data serves as a standard against which current image data is compared. As described herein above with respect to operation 630 of method 600, the estimation transforms the color balanced image data into multiple color spaces, extracts features from the transformed image data, and then for machine learning regression. Can be performed by generating a bilirubin estimate. At least some of the regressions described herein may also use some or all of the features extracted from the baseline skin color data.

  FIG. 11B illustrates a method 1150 for generating baseline skin color data, in accordance with many embodiments. In act 1160, baseline image data for a patient is captured when the patient has a reference bilirubin level, and the baseline image data corresponds to at least one image that includes a skin area and a baseline color calibration target. Acquisition of baseline image data may be similar to the acquisition procedure described herein above with respect to act 610 of method 600. The baseline color calibration target can be identical to the color calibration target used to collect normal image data, or can be a different color calibration target. Similarly, baseline image data can include the same skin area as normal image data or a different skin area. As mentioned above, the reference bilirubin level can be any known bilirubin level, such as bilirubin level, determined or photographed by testing within 24 hours of birth.

  In act 1170, color balancing baseline image data is generated for the skin area based on the baseline image data. The generation of color balancing baseline image data can be performed using any of the techniques described herein above with respect to regular image data. In act 1180, baseline skin color data is generated based on color balancing baseline image data for the skin area. The process may involve feature extraction from color balancing baseline image data and the use of features for machine learning regression, as described above.

  FIG. 12 illustrates a mobile device 1200 for estimating bilirubin levels in a patient, in accordance with many embodiments. Mobile device 1200 includes a camera 1202 suitable for capturing image data, and a flash unit 1204 suitable for providing flash illumination. Camera 1202 and flash unit 1204 may be embedded hardware of mobile device 1200. Alternatively, camera 1202 and flash unit 1204 can be provided separately from mobile device 1200 and coupled thereto (eg, via wired or wireless communication). In some cases, the mobile app described herein can detect whether the camera 1202 can capture an image with high enough resolution for subsequent image analysis, this criterion If not satisfied, the user can be alerted.

  Mobile device 1200 also includes an input unit 1206 for receiving input from a user, and a display 1208 for displaying content to the user. Input unit 1206 can include a keyboard, a mouse, a touch screen, a joystick, and the like. Input unit 1206 may also be configured to receive voice or gesture commands. Display 1208 can include a monitor, a screen, a touch screen, and the like. In some cases, input unit 1206 and display 1208 can be implemented across shared hardware (eg, the same touch screen is used to receive input and display output) . As mentioned above, the display 1208 can display one or more suitable UIs to the user, such as a mobile app UI for estimating bilirubin levels.

  Mobile device 1200 includes one or more processors 1210, a memory or other data storage device 1212 for storing image data as well as one or more software modules 1214, and a communication unit 1216. The processor 1210 is operatively coupled to the camera 1202 and / or the flash unit 1204 and can control one or more functions (eg, recording function, zoom function, flash illumination function). Processor 1210 can also be operatively coupled to memory 1212 such that processor 1210 can receive and execute instructions provided by software module 1214. Software module 1214 may be implemented as part of the mobile app described herein and may provide instructions to perform an operation of one or more of the aforementioned methods. For example, software module 1214 may enable mobile device 1200 to capture image data of a patient and a calibration target. In some cases, software module 1214 may perform some or all of the image processing and analysis tasks (eg, color balancing, feature extraction, machine learning regression) disclosed above. Software module 1214 may be adapted to multiple different types of mobile devices. In addition, mobile device 1200 receives and installs software updates, such as updates that improve one or more image capture, processing, and analysis algorithms, whereby the mobile app can easily and quickly, as needed. Can be configured to allow it to be upgraded.

  Communication unit 1216 of mobile device 1200 communicates data (eg, image data, bilirubin estimates, software updates, etc.) between mobile device 1200 and a separate device or system, such as a remote server or other computing system. It can be configured to receive and / or transmit. The communication unit may use any suitable combination of wired or wireless communication methods, including Wi-Fi communication. In some cases, communication between mobile device 1200 and a separate device may be performed using short message service (SMS) text messaging. Communication unit 1216 may also be operatively coupled to processor 1210 such that data communication to and from mobile device 1200 may be controlled based on instructions provided by software module 1214.

  FIG. 13 illustrates a mobile device 1300 in communication with a data processing system 1302 for estimating bilirubin levels, in accordance with many embodiments. Mobile device 1300 may include any of the components described herein above with respect to mobile device 1200 of FIG. Components of data processing system 1302 traverse any suitable combination of physical and / or virtualized computing resources (eg, virtual machines), including distributed computing resources (“in the cloud”). Can be implemented. In many embodiments, data processing system 1302 is a remote server configured to communicate with multiple user mobile devices, including mobile device 1300. Communication may utilize any suitable wired or wireless communication method, as described above.

  Data processing system 1302 includes one or more processors 1304, a memory or other data storage device 1306 for storing one or more software modules 1308, and a communication unit 1310. Communication unit 1310 may be used to communicate data (eg, image data, bilirubin estimates, software updates, etc.) between system 1302 and mobile device 1300 (eg, via SMS text messaging) it can. For example, communication unit 1310 can receive image data provided by mobile device 1300, such as non-color balanced image data. Data obtained from mobile device 1300 can be stored in memory 1306. The software module 1308 may process and analyze the image data (eg, color balancing, feature extraction, machine learning regression), such as by performing one or more of the methods described herein. , Instructions executable by processor 1304 may be provided. Processor 1304 can output an estimate of the bilirubin level of the patient, which can be stored in memory 1306 and / or transmitted to mobile device 1300. In some cases, depending on user preferences, the results may also be transmitted to a third party such as a healthcare professional who may review the results and provide further instructions to the user as needed. .

  In an alternative embodiment, the mobile device described herein illuminates the patient's skin with different wavelengths of light (eg, 460 nm, 540 nm) and uses a camera to image the irradiated skin It can be configured to capture. The timing and sequence of illumination can be controlled by the mobile app. The mobile app can analyze the collected image data and measure the intensity of the different light wavelengths reflected from the skin area. In many embodiments, the absorption of different wavelengths differs based on the color of the skin, including yellowing. Several wavelengths can be influenced by bilirubin levels, as their intensity provides an indication of the amount of bilirubin in the patient's body. Thus, a suitable machine learning regression and / or model can be developed to allow wavelength absorption to be used as an input to estimate bilirubin levels in a patient. Advantageously, the present approach can be more robust to different environmental and context conditions.

  In many embodiments, the mobile device includes a front camera (eg, a camera located on the same side as the screen and the screen of the mobile device), which is used to assess ambient lighting conditions. It can be used to capture image data (eg photos, videos). Such data provides alternative and / or additional ambient lighting information that can be used to normalize patient skin color data during estimation of patient bilirubin levels.

  The various techniques described herein may be implemented in part using code that can be stored on storage media and computer readable media and can be executed by one or more processors of a computer system. Or may be fully implemented. The storage medium and computer readable medium for containing the code or part of the code include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital multi application disc (DVD) or other optical storage, Magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, solid state drives (SSDs) or other solid state storage devices, or can be used to store desired information and can be accessed by system devices Implemented in any method or technique for storage and / or transmission of information such as computer readable instructions, data structures, program modules, or other data including any other medium, including but not limited to, volatilization And nonvolatility removability and Including storage media and communication media such as a removable medium, it can include any suitable medium that is known or used in the art. Based on the disclosures and teachings provided herein, one of ordinary skill in the art will recognize other approaches and / or methods for implementing the various embodiments.

  While preferred embodiments of the present invention are illustrated and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many variations, modifications, and substitutions will now occur to those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in practicing the invention. The following claims define the scope of the present invention, and methods and structures within the scope of these claims and their equivalents are intended to be covered thereby.

Claims (26)

A method of estimating the level of bilirubin in a patient, said method comprising
Receiving image data for at least one image including an area of skin of the patient and a color calibration target;
Generating color balanced image data for the skin area based on the color calibration target and the subset of the image data corresponding to the skin area, the subset of image data comprising at least one area of the image Including,
Estimating bilirubin levels in the patient based on color balanced image data for the skin area.
Estimating the bilirubin level is
Selecting a first estimated range of the bilirubin level from one of a plurality of different bilirubin ranges by processing a plurality of normalized chromatic and achromatic features;
Generating a final estimate of the bilirubin level by processing the plurality of normalized chromatic and achromatic features using an approach based on the selected first estimation range of the bilirubin level And how.   The method further includes receiving baseline skin color data for the patient corresponding to the patient having a reference bilirubin level, wherein estimating the bilirubin level comprises: comparing the baseline skin color data with the color for the skin area The method of claim 1, wherein the method is based on one or more differences between the balanced image data. Baseline skin color data for the patient is
Capturing baseline image data for the patient when the patient has the reference bilirubin level, wherein the baseline image data corresponds to at least one image including the skin area and a baseline color calibration target To do,
Generating color balancing baseline image data for the skin area based on the baseline color calibration target and the subset of the baseline image data corresponding to the skin area, the subset of the baseline image data Includes at least one region of the image, and
3. The method of claim 2, wherein the baseline skin color data is generated based on color balancing baseline image data for the skin area.   The method of claim 1, wherein the color calibration target comprises a plurality of standardized color regions including white regions.   The method according to claim 4, wherein the standardized color area comprises a black area, a gray area, a light brown area, a cyan area, a magenta area, a yellow area and a dark brown area.   The color calibration target is at least partially defined an opening, the opening being configured to enable capture of image data for the skin area by exposing the skin area. The method according to item 4.   The method according to claim 6, wherein the standardized color areas are arranged in a known arrangement surrounding the opening. Generating color balanced image data for the skin area is:
Identifying the subset of image data corresponding to the exposed skin area and the subset of image data corresponding to the white area by processing the received image data;
Determining the observed color value for the white area by processing the image data corresponding to the white area;
Generating color balanced image data for the exposed skin area based on observed color values for the white area.   9. The method of claim 8, wherein the observed color values for the white area comprise red, green, blue (RGB) color space values. The color balanced image data for the skin area comprises RGB color space data, and the method comprises converting the RGB color space data to at least one other color space to obtain the at least one other color space. 7. The method of claim 6 , further comprising generating color balanced image data for the exposed skin area for.   The at least one other color space comprises (a) a cyan, magenta, yellow and black (CMYK) color space, (b) a YCbCr color space, or (c) a Lab color space. Method described.   11. The method of claim 10, wherein the at least one other color space comprises (a) a cyan, magenta, yellow (CMY) color space, (b) a YCbCr color space, or (c) a Lab color space. Method.   The method according to claim 11, wherein the received image data comprises an image obtained using flash illumination and an image obtained without flash illumination.   The method of claim 1, wherein the plurality of different bilirubin ranges include low range, mid range, high range.   The method of claim 1, wherein the plurality of normalized chromatic and achromatic features comprises selected color values of the skin area for a plurality of different color spaces.   The method of claim 1, wherein the plurality of normalized chromatic and achromatic features include the calculation of a color gradient across the skin area.   Selecting the first estimation range of the bilirubin level by processing the plurality of normalized chromatic and achromatic features includes (a) linear regression, (b) encapsulated k nearest neighbor regression, (C) Lasso regression, (d) LARS regression, (e) elastic net regression, (f) support vector regression using linear kernels, (g) support vector regression to assign higher weights to higher bilirubin values, The method of claim 1, comprising performing a series of regressions comprising at least one of (h) random forest regressions.   Selecting the first estimated range of the bilirubin level by processing the plurality of normalized chromatic and achromatic features comprises: (a) linear regression, encapsulation k nearest neighbor regression, (c) Lasso regression, (d) LARS regression, (e) elastic net regression, (f) support vector regression using linear kernels, (g) support vector regression, which assigns higher weights to higher rates of bilirubin values, (h) The method of claim 1, comprising performing a series of regressions, including random forest regressions.   Using a processing approach based on the selected first estimation range of the bilirubin level may include, as features for a final random forest regression, the plurality of normalized chromatic and achromatic features and the bilirubin level. The method of claim 1, comprising performing a final random forest regression using the selected first estimation range of.   Estimating the bilirubin level comprises: determining a color space value for the skin of the patient; and estimating the bilirubin level using a processing approach based on the color space value of the skin of the patient determined. The method of claim 1, comprising: A mobile device configured to estimate the level of bilirubin in a patient, the mobile device comprising
A camera operable to capture image data for a field of view;
A processor operatively coupled to the camera;
A data storage device operatively coupled to the processor, the data storage device storing instructions.
The instructions are executed by the processor:
Receiving image data for an image captured by the camera, wherein the image includes an area of skin of the patient and a color calibration target.
Generating color balanced image data for the skin area based on the color calibration target and the subset of image data corresponding to the skin area, the subset of image data including at least one area of the image , And
Estimating the bilirubin level in the patient based on color balanced image data for the skin area.
Estimating the bilirubin level is
Selecting a first estimated range of the bilirubin level from one of a plurality of different bilirubin ranges by processing a plurality of normalized chromatic and achromatic features;
Generating a final estimate of the bilirubin level by processing the plurality of normalized chromatic and achromatic features using an approach based on the selected first estimation range of the bilirubin level Mobile device, done by and. The color calibration target at least partially defines an opening, the opening being configured to enable capturing image data for the skin area by exposing the skin area. Wherein said color calibration target comprises a plurality of standardized color areas including a white area;
The instruction is
Identifying the subset of image data corresponding to the exposed skin area and the subset of image data corresponding to the white area by processing the received image data;
Determining the observed color value for the white area by processing the image data corresponding to the white area;
Generating color balanced RGB image data for the exposed skin area based on the observed color values for the white area;
Generating color-balanced image data for the exposed skin area for at least one other color space by converting the color-balanced RGB image data to at least one other color space. 22. The mobile device of claim 21, having a processor. The color calibration target at least partially defines an opening, the opening being configured to enable capturing image data for the skin area by exposing the skin area. Wherein said color calibration target comprises a plurality of standardized color areas including a white area;
The instruction is
Identifying the subset of image data corresponding to the exposed skin area and the subset of image data corresponding to the white area by processing the received image data;
Determining the observed color value for the white area by processing the image data corresponding to the white area;
Generating color balanced RGB image data for the exposed skin area based on the observed color values for the white area;
Generating color balanced image data for the exposed skin area for at least one other color space by converting the color balancing RGB image data to at least one other color space;
Determining skin color for the patient by processing color balanced image data for the exposed skin area;
22. The processor comprising: estimating the bilirubin level by processing a plurality of normalized chromatic and achromatic features using the determined skin color based approach. Described mobile device.   The apparatus further comprises a flash unit operable to selectively illuminate the field of view, wherein received image data processed to estimate the bilirubin level is captured using the field of view illuminated by the flash unit. 24. A mobile device according to any of claims 22 and 23, comprising an image and an image captured using a field of view not illuminated by the flash unit. A method for estimating bilirubin levels in a patient, said method comprising
Receiving image data for an image from the mobile device, including the patient's skin area and the color calibration target;
Generating color balanced image data for the skin area based on the color calibration target and a subset of the image data corresponding to the skin area via one or more processors, the subset of the image data Includes at least one region of the image, and
Estimating bilirubin levels in the patient based on color balanced image data for the skin area via the one or more processors;
Transmitting the estimated bilirubin level to the mobile device.
Estimating the bilirubin level is
Selecting a first estimated range of the bilirubin level from one of a plurality of different bilirubin ranges by processing a plurality of normalized chromatic and achromatic features through the one or more processors And
Processing the plurality of normalized chromatic and achromatic features using the selected first estimated range-based approach of the bilirubin level via the one or more processors; Generating a final estimate of bilirubin level.   26. The method of claim 25, wherein at least one of receiving the image data and transmitting the estimated bilirubin level is performed using short message service (SMS) text messaging.

Images