JP7542578B2 - Methods and systems for utilizing quantitative imaging - Patents.com - Google Patents

Description

The present disclosure relates to quantitative imaging and analysis. More specifically, the present disclosure relates to systems and methods for analyzing lesions using quantitative imaging.

Imaging, especially with safe, non-invasive methods, is the most powerful way to identify the cause of disease, characterize its detailed pathology, prescribe therapy, and monitor progression to health. Imaging is also an extremely valuable, low-cost method to mitigate these human and economic costs by enabling appropriate early interventions that are less disruptive and less costly.

Enhanced imaging technologies have made medical imaging an essential component of patient care. Imaging is particularly valuable because it provides spatially and temporally localized anatomical and functional information using non-invasive or minimally invasive methods. However, techniques are needed to effectively utilize the increased spatial and temporal resolution to exploit patterns or features in the data that are not easily assessed by the human eye, and to manage large amounts of data in a manner that efficiently integrates the data into clinical workflow. Without assistance, clinicians often lack the time and often the ability to effectively extract the available information content, and in any case typically interpret the information subjectively and qualitatively. Integrating quantitative imaging for individual patient management and clinical trials for therapy development calls for a new class of decision support informatics tools that enable the medical community to fully exploit the capabilities possible with evolving and growing imaging modalities within the constraints of existing workflows and reimbursement.

Quantitative results from imaging methods may be used as biomarkers in both routine clinical care and clinical trials, for example, according to the widely accepted NIH Consensus Conference definition of biomarkers. In clinical practice, quantitative imaging aims to (a) detect and characterize disease before, during, or after a course of therapy, and (b) predict disease course with or without therapy. In clinical research, imaging biomarkers can be used to define clinical trial endpoints.

Quantification is based on improvements in spatial, temporal, and contrast resolution, as well as developments in imaging physics that allow tissues to be excited with multiple energies/sequences, resulting in diverse tissue-specific responses. These improvements allow tissue identification and functional assessment, as seen, for example, in computed tomography (CT), dual-energy computed tomography (DECT), spectral computed tomography (spectral CT), computed tomography angiography (CTA), cardiac computed tomography angiography (CCTA), magnetic resonance imaging (MRI), multi-contrast magnetic resonance imaging (multi-contrast MRI), ultrasound (US), and targeted or general contrast approaches using various imaging modalities. Quantitative imaging measures specific biological properties that indicate the effectiveness of one treatment over another, how effective a current treatment is, or what risks a patient faces if they remain untreated. Viewed as a measuring device, the scanner, combined with image processing of the formed images, has the ability to measure tissue properties based on physical principles related to a given imaging approach and how different tissues respond to it. While the imaging process varies widely across modalities, and some generalizations can be helpful in framing the overall evaluation, there are exceptions, nuances, and subtleties that will drive real conclusions, and until and unless they are taken into account, some of the greatest opportunities are missed.

Imaging in early phases of clinical trials of novel therapeutics contributes to the understanding of underlying biological pathways and pharmacological effects. Imaging may also reduce the cost and time required to develop new medicines and therapies. In later phases of development, imaging biomarkers may serve as critical endpoints for clinical benefit and/or as companion diagnostics to help describe and/or track a particular patient's status for personalized therapy. In all phases, imaging biomarkers can be used to select or stratify patients based on disease status to better demonstrate the efficacy of therapy.

a. Quantitative Medical Imaging Enhanced imaging technology has made medical imaging an essential component of patient care. Imaging is particularly valuable because it provides spatially and temporally localized anatomical and functional information using non-invasive or minimally invasive methods. However, techniques for improved resolution are needed to exploit patterns or features in the data that are not easily assessed by the human eye, and to manage large amounts of data in a way that efficiently integrates the data into clinical workflow. With modern high-resolution imaging techniques, radiologists are left "drowning" in data without assistance. Integrating quantitative imaging for individual patient management will require a new class of decision support informatics tools to enable the community to fully exploit the capabilities of these new tools within existing workflow and reimbursement constraints.

Additionally, quantitative imaging methods are becoming increasingly important for (i) preclinical trials, (ii) clinical research, (iii) clinical trials, and (iv) clinical practice. Imaging in early phases of clinical trials of novel therapeutics contributes to the understanding of underlying biological pathways and pharmacological effects. Imaging may also reduce the cost and time required to develop novel drugs and therapies. In later phases of development, imaging biomarkers may serve as important endpoints for clinical benefit. In all phases, imaging biomarkers can be used to select or stratify patients based on disease status to better demonstrate the efficacy of therapies.

Improved patient selection through the use of quantitative imaging may reduce the sample size required for a given trial (by increasing the proportion of evaluable patients and/or reducing the influence of nuisance variables) and help identify subpopulations that are most likely to benefit from a proposed therapy. This should reduce drug development time and costs, but may also correspondingly reduce the size of the "target" population.

Diseases are not simple, and while disease often manifests locally, it is often systemic. Multifactorial assessment of objectively relevant tissue characteristics, expressed as a panel or "profile" of continuous indicators, ideally suited as "surrogates" for sometimes prospective and/or difficult to measure but accepted endpoints, has proven to be an effective method across medicine, and will be followed here. Computer-assisted measurement of lesion and/or organ biology, as well as quantification of tissue composition in a first-reader or second-reader paradigm, enabled by the interdisciplinary convergence of next-generation computational methods for personalized diagnosis based on quantitative imaging assessment of phenotype, implemented in an architecture that actively optimizes interoperability with modern clinical IT systems, will empower clinicians in managing patients with disease across the severity continuum, improving patient stratification across surgical, medical, and monitoring pathways. More timely and accurate assessments will yield improved outcomes, more efficient use of healthcare resources, benefits that far outweigh the cost of the tools, at a level of granularity and sophistication that approaches the complexity of the disease itself, rather than holding assumptions that they can be simplified to a level that is at odds with the underlying biology.

b. Phenotype:
Radiological imaging is generally interpreted subjectively and qualitatively with respect to disease states. Medical literature uses the term phenotype as the set of observable characteristics of an individual resulting from the interaction of its genotype with the environment. Phenotype generally implies objectivity, i.e., phenotype is factual rather than subjective. Radiology is well known for its ability to visualize characteristics that can, increasingly, be verified with objective factual standards (US Application Serial No. 14/959,732, US Application Serial No. 15/237,249, US Application Serial No. 15/874,474, and US Application Serial No. 16/102,042). As a result, while radiological images can be used to determine phenotypes, there is often a lack of suitable means to do so.

Advantageously, phenotyping is fact-based and can be assessed independently and objectively. Furthermore, phenotyping is a format already accepted by the healthcare industry to manage patient treatment decisions. Thus, phenotyping has a high degree of clinical relevance. Finally, phenotyping is relevant to consumers, allowing both self-advocacy and motivation for lifestyle changes.

Early identification of phenotypes based not only on detection of single features but on a comprehensive panel of continuous indicators would allow for rapid intervention to prevent irreversible damage and death. Solutions are essential to get ahead of the event entirely, or at least improve diagnostic accuracy when signs and/or symptoms are experienced. Efficient workflow solutions with automated measurements of structure and quantification of tissue composition and/or hemodynamics could be used to characterize high-risk patients (who would end up receiving different treatment than non-high-risk patients). Linking plaque morphology characteristics to embolic potential would have major clinical implications.

Imaging phenotypes can be gene expression patterns that are correlated in association studies. Imaging phenotypes can have clinical impact as imaging is routinely used in clinical settings, providing unprecedented opportunities to improve decision support in personalized treatment at low cost. Correlating specific imaging phenotypes with large-scale genomic, proteomic, transcriptomic, and/or metabolomic analyses may impact therapy strategies by creating more deterministic and patient-specific prognosis and measures of response to drug or other therapies. However, methods to extract imaging phenotypes to date are mostly empirical in nature and based primarily on human, albeit expert, observation. These methods are subject to human variability and are clearly not scalable to support high-throughput analysis.

At the same time, the convergence of unmet needs to enable more personalized care without adding cost -- in effect, through initiatives in preventive care, comparative effectiveness, reimbursement approaches, and/or avoiding rather than reacting to unexpected events -- for better cost control is putting unprecedented pressure on technological advances that not only provide the capability, but also provide it in ways that simultaneously reduce costs.

In addition to the problem of phenotyping (classification into unordered categories), there is the problem of outcome prediction/risk stratification (prediction of normal levels of risk). Both have clinical utility, but the technical device characteristics result in different ones. Specifically, one is not strictly dependent on the other.

Without limiting generality, examples of phenotypic classifications with clinical relevance are given below:

Exemplary symptoms of the "stable plaque" phenotype of arteriosclerosis can be described as follows.
• "Cure" disease, poor response to intensive antistatin regimes • Fewer balloon/stent complications • Stenosis can be >50% • Ca can be high and deep • Minimal or no lipid, bleeding or ulceration • Smooth appearance Such plaques are generally associated with a lower incidence of adverse events than the "unstable plaque" phenotype.
● Active disease, may be highly responsive to lipid lowering and/or intensified anti-inflammatory regime ● Common balloon/stent complications ● Stenosis may be below 50% ● Low or diffuse Ca, proximal Ca, napkin ring sign, and/or microcalcifications ● May have evidence of high lipid content, thin cap ● May have evidence of hemorrhage or intraplaque hemorrhage (IPH), and/or ulceration

Such plaques have been reported to have a three- to four-fold higher incidence of adverse events compared to stable phenotypes. While these two examples may be assessed in a single patient encounter, other phenotypes, such as "rapidly progressive," may be determined by comparing the rate of change of a characteristic over time; that is, a phenotype determined not just based on what is statistically present at one point in time, but rather on the kinematics and/or how it is changing n times.

c. Machine and Deep Learning Technologies:
Deep learning (DL) methods have been applied with great success to many challenging machine learning (ML) and classification tasks resulting from complex real-world problems. Notable recent applications include computer vision (optical character recognition, face recognition, satellite image interpretation, etc.), speech recognition, natural language processing, medical image analysis (image segmentation, feature extraction, and classification), and biomarker discovery and validation in clinical and molecular data. An attractive feature of this approach is its applicability to both unsupervised and supervised learning tasks.

Neural networks (NNs) and deep NN approaches, which broadly include convolutional neural networks (CNNs), recurrent convolutional neural networks (RCNNs), etc., have been shown to be based on a sound theoretical foundation and are extensively modeled on principles believed to represent advanced cognitive functions of the human brain. For example, in the neocortex, a brain region associated with many cognitive abilities, sensory signals propagate through complex local modular hierarchies that learn to represent observations over time. This design principle has led to the popular definition and construction of CNNs used for image classification and feature extraction. However, it remains a somewhat open question what the more fundamental reasons are for the superior performance of DL networks and approaches compared to frameworks that have the same number of fitting parameters but do not have a deep architecture.

Conventional ML approaches to image feature extraction from raw data and image-based learning have many limitations. In particular, when features are at a summary level rather than the 3D level of the original image, or at the 3D level, ML approaches using feature extraction often have difficulty capturing spatial context to the extent that they are not bound to objectively verifiable biological facts and are merely mathematical operators. While the use of raw datasets with spatial context often lacks objective ground truth labels for the extracted features, the use of raw image data itself contains a lot of variation that is "noise" with respect to the classification problem at hand. In applications other than imaging, this variation is typically mitigated by very large training sets, so that the network training "learns" to ignore this variation and focus only on the important information, but large training sets of this magnitude are generally unavailable for medical applications, especially for ground truth annotated datasets, due to cost, logistics, and privacy issues. The systems and methods of the present disclosure help overcome these limitations.

d. Exemplary Applications: Cardiovascular measurements such as Fractional Flow Reserve (FFR) and Plaque Stability or High Risk Plaque (HRP):
Although new treatments have been revolutionary in improving outcomes over the past 30 years, cardiovascular disease still costs the US economy $320 billion annually. There is a significant patient population that could benefit from better characterization of risk of major coronary or cerebral adverse events. The American Heart Association (AHA) estimates that 9.4% of adults (>20 years) have a risk of >20% adverse events in the next 10 years, and 26% have a risk of 7.5%-20%, based on population extrapolation of ACSD (atherosclerotic cardiovascular disease) risk scores. Applying this to the population, we get 23 million high-risk patients and 57 million moderate-risk patients. The 80 million at risk can be compared to the 30 million US patients currently taking statin therapy to avoid new or recurrent events, and the 16.5 million with a CVD diagnosis. Some patients taking statins develop obstructive disease and acute coronary syndromes (ACS). The majority of patients are unaware of disease progression until they develop chest pain. Further improvements in outcomes and costs of coronary artery disease will flow from improved non-invasive diagnostics and identifying which patients have disease that is progressing under primary care.

Heart health can be greatly affected by the deterioration of the arteries surrounding the heart. Various factors (e.g., angiogenesis, neovascularization, inflammation, calcification, lipid deposition, necrosis, hemorrhage, stiffness, density, stenosis, dilation, remodeling ratio, ulceration, flow (e.g., of blood in channels), pressure (e.g., of blood in channels or one tissue pressing against another), cell type (e.g., macrophages), cell alignment (e.g., of smooth muscle cells), or tissue properties such as shear stress (e.g., of blood in channels), cap thickness, and/or tortuosity (e.g., inlet and outlet angles)) can cause these arteries to become less effective at transmitting oxygen-filled blood to the surrounding tissues (Figure 35).

Coronary artery function testing, primarily stress electrocardiography and single photon emission computed tomography myocardial perfusion testing (SPECT MPI), are the primary noninvasive methods for the diagnosis of obstructive coronary artery disease. Over 10 million function tests are performed annually in the United States, and positive results prompt 2.6 million catheterization laboratory visits for invasive angiograms to confirm findings of coronary artery disease.

Another approach to assessing perfusion is to determine the ability of the vasculature to deliver oxygen. Specifically, the decline in ability can be quantified as the fractional flow reserve, or FFR. FFR is not a direct measure of ischemia, but rather a surrogate that measures the ratio of pressure drop across a lesion. The change in luminal diameter relative to other segments of the same vessel caused by local vasodilation impairment during maximal hyperemia produces significant hemodynamic effects that lead to abnormal FFR measurements. During physical FFR measurements, the injection of adenosine reduces downstream resistance, thereby increasing flow in hyperemic conditions. To physically measure FFR, an invasive surgical procedure involving a physical pressure sensor in the artery is required. This level of invasiveness adds risk and inconvenience, so there is a need for a method to estimate FFR with high accuracy without the need for physical measurements. The ability to perform this measurement noninvasively also reduces the noted "treatment bias." Because stenting is relatively easy to do once the patient is in the catheterization lab, the ability to noninvasively assess fractional flow reserve may improve the decision to stent, which many have noted leads to overtreatment. Similarly, fractional flow reserve also applies to brain tissue perfusion (e.g., in relation to cerebral congestion).

Functional testing has known problems with sensitivity and specificity. It is estimated that approximately 30-50% of patients with cardiovascular disease are misclassified and over- or under-treated, resulting in significant costs in monetary and quality of life. Functional testing is also expensive and time-consuming, limiting its use in patients with pre-obstructive disease. False positives from non-invasive functional testing are a major contributor to the overuse of both invasive coronary angiography and percutaneous coronary intervention in stable patients and are a major policy issue in the United States, the United Kingdom, and China. Studies of the impact of false negatives have estimated that of the 3.8 million MPI tests performed annually in patients with suspected coronary artery disease (CAD) in the United States, nearly 600,000 of these report false-negative findings, leading to 13,700 acute coronary events, many of which could be prevented simply by introducing appropriate medical therapy. Another drawback of functional testing is that it is transient in nature. Ischemia is a lagging indicator that follows anatomical changes caused by disease progression. Patients at high risk for ACS would be better served if a future causative lesion could be detected and mitigated with intensive medical therapy before the onset of ischemia.

Coronary computed tomography angiography (CCTA), especially when used in conjunction with quantitative analysis software, is evolving to become the ideal test to fill this gap in understanding the extent and rate of progression of coronary artery disease. Over the past decade, CT scanners in most countries have been upgraded to faster, higher detector count machines that can provide superior spatial resolution without slowing the heart or requiring breath holding. Radiation doses have been significantly reduced to equivalent or less than SPECT MPI and invasive angiograms.

Recent analyses of data from landmark trials such as SCOT-HEART, PREDICT, and PROMISE have demonstrated the value of using CCTA to detect non-obstructive disease, sometimes called high-risk plaque (HRP) or vulnerable plaque, by identifying patients (those at high risk for future adverse events). Study designs were diverse and included nested case-control cohorts comparing CCTA-enrolled patients with cardiovascular (CV) events to controls with similar risk factors/demographics, comparisons with FFR, and multiyear follow-up to large-scale "test and treat" studies. The recent favorable decision from NICE to position CCTA as the primary diagnosis was based on a significant reduction in CV events attributable to initiation of drug therapy or change in plaque detection by CCTA in the CCTA arm of the SCOT-HEART study.

Important target patient groups are patients with stable chest pain and no history of CAD with typical or atypical angina symptoms (based on SCOT-HEART data), as well as patients with non-obstructive disease (stenosis <70%) and younger patients (e.g., 50-65 years old group) based on PROMISE findings suggesting that plaque evaluation is most necessary. Patients with non-obstructive CAD with a high-risk plaque profile based on CCTA analysis can be assigned to the most appropriate high-intensity statin therapy (especially when considering decisions regarding very expensive new lipid-lowering therapies such as PCSK9 inhibitors, or anti-inflammatory drugs such as canakinumab), or new antiplatelet agents to reduce the risk of coronary thrombosis, and/or additional long-term follow-up with the possibility of intensifying or downgrading therapy. CCTA is an ideal diagnostic tool because it is non-invasive and requires less radiation than cardiac catheterization.

Pathology literature on culprit lesions associated with fatal heart attacks has noted that clinically nonobstructive CAD is much more likely to harbor the highest risk plaques than more obstructive plaques, which tend to be more stable. These findings were supported by a recent study that looked at culprit lesions from ACS patients undergoing invasive angiography and compared these to baseline CCTA precursor plaques. In one cohort undergoing clinically indicated CCTA, for patients found to have nonobstructive CAD, 38% of patients so examined still had a significant risk of mid- to long-term major adverse cardiovascular and/or cerebrovascular events (MACCE). Hazard ratios based on the number of affected segments, independent of obstruction, were found to be significant long-term predictors of MACCE in this group. One factor contributing to the predictive value of clinically nonobstructive CAD is that these lesions are much more likely to harbor the highest risk plaques than more obstructive plaques, which tend to be more stable.

The potential utility of CCTA in detecting and managing occlusive and preocclusive atherosclerotic lesions has also been seen in several recently published longitudinal studies of the therapeutic effects of statins and anti-inflammatory drugs, where plaque remodeling to a more stable state and plaque regression were observed in the treatment group. This supports studies that have investigated disease progression and treatment effects under various lipid-reducing drug protocols, including early intravascular ultrasound (IVUS, sometimes with "virtual histology" VH), near-infrared spectroscopy (NIRS), and optical coherence tomography (OCT). Recent clinical trials provide potential plaque biomarkers to demonstrate the efficacy of new medical therapies. The Integrated Biomarkers and Imaging Study-4 (IBIS-4) found progression of calcification as a potential protective effect of statins. Other studies found that lipid-rich necrotic core (LRNC) decreased under statin treatment. In these studies, clinical variables used alone were not discriminatory enough to identify high-risk plaque characteristics. These studies highlight the importance of complete characterization and evaluation of the entire coronary tree, not just the culprit lesion, to allow for more accurate risk stratification, which CCTA can favorably analyze. In a meta-analysis, CCTA had a superior diagnostic accuracy for detecting coronary plaque compared with IVUS, but there were small differences in the assessment of plaque area and volume, percentage of stenosis, and slight overestimation of lumen area. Adding the percentage change of lipid-rich necrotic core and its distance from the lumen was also found to have high prognostic value. Additionally, the results of the ROMICAT II trial show that identifying high-risk plaque with CCTA in stable CAD patients with acute chest pain but negative initial ECG and troponin increases the likelihood of significant CAD and ACS independent of clinical risk assessment. CCTA is an established test for the assessment of coronary atherosclerotic plaque. For patients with uncertain need for invasive procedures, predicting MACCE noninvasively would be beneficial and feasible with CCTA providing a global estimate of disease burden and risk of future events.

The prevalence of carotid artery disease and CAD are closely related. Carotid atherosclerosis has been shown to be an independent predictor of MACCE, even in patients without pre-existing CAD. Such findings suggest a common underlying etiology shared by both conditions, which is further supported by the Multi-Ethnic Study of Atherosclerosis (MESA). Atherosclerosis develops progressively through the development of lesions in the arterial wall as a result of cholesterol-rich lipid accumulation and inflammatory responses. These changes are similar (if not identical) in coronary, carotid, aortic, and peripheral arteries. Certain plaque characteristics, such as a large atherosclerotic core with lipid-rich contents, a thin cap, outward remodeling, infiltration of the plaque by macrophages and lymphocytes, and media thinning, predispose to vulnerability and rupture.

e. Non-invasive determination of HRP and/or FFR:
Non-invasive evaluation of the functional significance of stenosis using CCTA is of clinical and economic interest. The combination of the contour or anatomy of the lesion or vessel and the properties or composition of the tissues, including the wall and/or plaques in the wall, collectively referred to as plaque morphology, can explain the outcome of the lesion by orthogonal consideration of high-risk or low-risk plaques (HRP), and/or normal and abnormal fractional flow reserve (FFR). Lesions with large necrotic cores may develop dynamic stenosis due to outward remodeling during plaque formation, resulting in more tissue stretching, tissue stiffening, or the smooth muscle layer may already be stretched to the limit of the Glagov phenomenon, after which the lesion may erode the lumen itself. Similarly, inflammatory injury and/or oxidative stress may result in local endothelial dysfunction, manifesting as a reduced vasodilatory capacity.

When the tissue that makes up the plaque is matrix or "fatty streaks" that are not organized into a necrotic core, the plaque expands sufficiently to keep up with demand. However, when the plaque has a more substantial necrotic core, it cannot expand; the blood supply cannot keep up with demand. Plaque morphology improves accuracy by assessing complicating factors such as LRNC, calcification, hemorrhage, and ulceration with objective facts that can be used to validate the underlying information in ways that other approaches cannot due to the lack of objective validation of intermediate measurements.

But that's not all plaque can do. All too often, plaques rupture and suddenly cause a thrombus that then causes an infarction of the heart or brain tissue. Plaque morphology also identifies and quantifies these high-risk plaques. For example, plaque morphology can be used to determine how close the necrotic core is to the lumen, which is a key determinant of infarction risk. Knowing whether a lesion will limit flow under stress does not indicate risk of rupture, and vice versa. Other methods, such as computational fluid dynamics (CFD), without objectively validated plaque morphology, can simulate flow limitation but not risk of infarction. The fundamental advantage of plaque morphology is that its precision lies in the determination of both vascular structure and tissue properties, which together allow for phenotyping.

Increasingly, clinical guidelines are available regarding the optimal management of patients with different assessments of fractional flow reserve. Obstructive lesions with high-risk features (large necrotic core and thin cap) are known to predict the greatest likelihood of future events, and importantly, the reverse is also true.

Approaches have been published that use CFD to determine FFR without accurately assessing plaque morphology. However, CFD-based fractional flow reserve considers only the lumen, or at best, how the lumen surface changes during different parts of the cardiac cycle. At best, considering only the lumen surface would require processing both systole and diastole to obtain motion vectors (which most available methods do not do), but still do not consider what happens during stress, since these analyses are done not by measuring actual stress, but by computer simulations of what happens under stress, and are based only on the blood flow path, not on the actual properties that give rise to vasodilation capacity. Some methods try to simulate forces and apply biomechanical models, but use assumptions, rather than validated measurements of wall tissue. As a result, these methods lack the ability to predict what may happen if stress actually exceeds rudimentary assumptions and causes rupture. Rather, characterizing the tissue solves these problems. Wall properties, including the influence on the vessel's vasodilation capacity due to wall distensibility, are considered superior in that lesion configuration determines flexibility and energy absorption, and stable lesions are still overtreated and MACCE risk is incompletely assessed. Advantages of using morphology to assess FFR include the fact that morphology is a leading indicator, FFR is lagging, and the presence and degree of HRP aids in the treatment of borderline subjects. The importance of resolving accurate assessment by morphology is reinforced by a growing number of studies showing that morphology can predict FFR, but FFR does not predict morphology. In other words, effectively assessed morphology has the ability to determine not only FFR but also the potential for discontinuous changes in plaque that move patients from ischemia to infarction or HRP.

Provided herein are systems and methods that utilize a hierarchical analytical framework to identify and quantify biological properties/analytes from imaging data, and then identify and characterize one or more disease states based on the quantified biological properties/analytes. In some embodiments, the systems and methods incorporate computerized image analysis and data fusion algorithms along with patient clinical chemistry and blood biomarker data to provide a multi-factorial panel that can be used to distinguish different subtypes of disease. Thus, the systems and methods of the present disclosure can advantageously implement biological and clinical insights into advanced computational models. These models can be coupled with sophisticated image processing through rich ontologies that identify factors associated with an increased understanding of disease etiology, taking the form of rigorous definitions of what is being measured, how it is being measured and evaluated, and how it is associated with clinically relevant subtypes and stages of disease against which it may be validated.

Human diseases exhibit strong phenotypic differences that can be understood by applying sophisticated classifiers to extracted features that capture the spatial, temporal, and spectral outcomes measurable by imaging but are difficult to understand in isolation. Traditional computer-aided diagnosis makes inferences in a single step from image features. In contrast, the disclosed systems and methods use a hierarchical inference scheme that includes intermediate steps of determining spatially resolved image features and temporally resolved dynamics at multiple levels of biologically objective components of morphology, composition, and structure, which are then utilized to guide clinical inferences. The hierarchical inference scheme advantageously ensures that clinical inferences are understood, verified, and explained at each level of the hierarchy.

Thus, in an exemplary embodiment, the disclosed systems and methods utilize a hierarchical analysis framework consisting of a first level of algorithms that measure biological properties that can be objectively verified against a standard of fact independent of imaging, and then a second set of algorithms to determine a medical or clinical condition based on the measured biological properties. This framework is applicable in an "and/or" fashion, i.e., alone or in combination, to several different biological properties, such as angiogenesis, neovascularization, inflammation, calcification, lipid deposition, necrosis, hemorrhage, stiffness, density, stenosis, dilation, remodeling ratio, ulceration, flow (e.g., of blood in a channel), pressure (e.g., of blood in a channel or of one tissue pressing against another), cell type (e.g., macrophages), cell alignment (e.g., of smooth muscle cells), or shear stress (e.g., of blood in a channel), cap thickness, and/or tortuosity (e.g., entrance and exit angles), etc. Each of these measurements can be measured in terms of quantity and/or degree and/or characteristics of the property. Examples of conditions include perfusion/ischemia (e.g., of brain or heart tissue) (as limited), perfusion/infarction (e.g., of brain or heart tissue) (as completely severed), oxygenation, metabolism, fractional flow reserve (ability to perfuse), malignancy, invasion, and/or risk stratification (as probability of event, or time to event (TTE)), e.g., major adverse cardiocerebrovascular events (MACCE). Factual evidence includes, for example, biopsy, expert tissue annotation from resected tissue (e.g., intra-arterial resection or autopsy), expert phenotypic annotation on resected tissue (e.g., intra-arterial resection or autopsy), physical pressure lines, other imaging methods, physiological monitoring (e.g., ECG, SaO2, etc.), genomic and/or proteomic and/or metabolomic and/or transcriptomic assays, and/or clinical outcomes. These characteristics and/or conditions may be assessed at a given time point and/or may change over time (longitudinal).

In an exemplary embodiment, the systems and methods of the present application are advantageously related to computer-aided phenotyping (CAP) of disease. CAP is a new and exciting complement to the field of computer-aided diagnosis (CAD). As disclosed herein, CAP may apply hierarchical reasoning incorporating computerized image analysis and data fusion algorithms to a patient's clinical chemistry and blood biomarker data to provide a multifactorial panel or "profile" of measurements that can be used to distinguish different subtypes of disease that would receive different treatments. Thus, CAP implements new approaches to robust feature extraction, data integration, and scalable computational strategies to implement clinical decision support. For example, spatiotemporal texture (SpTeT) methods capture relevant statistical features to spatially and kinetically characterize tissues. Spatial features are mapped to characteristic patterns of, for example, extracellular matrix fibers, necrotic tissue, and/or lipids mixed with inflammatory cells. The kinetic signatures map, for example, to endothelial permeability, angiogenesis, necrosis, and/or collagen degradation.

In contrast to current CAD approaches that perform clinical inference in a single step of machine classification from image features, the subject application system and method can advantageously apply a hierarchical inference scheme that starts from spatially resolved image features as well as multiple levels of temporally resolved dynamics of biologically objective components of intermediate morphology and structural composition, and finally performs clinical inference. This results in a system that can be understood, verified, and explained at each level of the hierarchy, from low-level image features at the bottom to biological and clinical features at the top.

The disclosed system and method improve both phenotyping and outcome prediction. Phenotyping can occur at two levels: individual anatomical locations on the one hand, and more generally described body parts on the other. The input data for the former can be 2D datasets, while the input data for the latter can be 3D datasets. In phenotyping, the objective facts can be at either level, whereas outcome prediction/risk stratification generally occurs at the patient level, but can be more specific in certain instances (e.g., which side the stroke symptoms appeared on). The implication here is that the same input data can be used for both purposes, but the models differ significantly depending on the level at which the input data is used and the basis of the annotation of facts.

While it is possible to implement model building reading vectors as input data, performance is often limited by the measurements implemented. The subject application advantageously utilizes unique measurements (e.g., cap thickness, calcium depth, and ulceration) to improve performance. Thus, a reading vector only approach can be applied (e.g., in combination with traditional measurements) when the vector includes these measurements. However, the systems and methods of the present disclosure can advantageously utilize deep learning (DL) approaches, which can provide even richer data sets. The subject application systems and methods can also advantageously utilize unsupervised learning applications, thereby providing better scalability across the data domain (a highly desirable feature keeping in mind the pace at which new biomedical data is generated).

In the exemplary embodiment presented herein, a convolutional neural network (CNN) can be utilized to build a classifier in an approach that can be characterized as transfer learning with fine-tuning approach. A CNN trained on a large compendium of imaging data on a powerful computational platform can be used to classify images that were not annotated in the network training. This makes intuitive sense because many common class features are useful for discriminating images of very different objects (i.e., shape, boundary, spatial orientation, etc.). In that case, a CNN trained to recognize thousands of different objects using a pre-annotated dataset of tens of millions of images can conceivably perform basic image recognition tasks much better than chance, and with relatively little fine-tuning of the final classification layer (sometimes called a "softmax layer"), it can perform comparable to a CNN trained from scratch. Because these models are so large and trained on a huge number of pre-annotated images, they tend to "learn" imaging features that are highly distinctive and discriminatory. Thus, convolutional layers can be used as feature extractors, or already trained convolutional layers can be fine-tuned to fit the problem at hand. The former approach is called transfer learning, the latter fine-tuning.

CNNs excel at performing a variety of computer vision tasks. However, CNNs have several drawbacks. Two key drawbacks that are important for medical systems are 1) the need for huge training and validation datasets, and 2) the intermediate CNN computations do not represent any measurable properties (sometimes criticized as "black boxes" whose rationale is not explained). The approach disclosed herein may advantageously utilize a pipeline consisting of one or more biologically measurable and independently verifiable stages, followed by a convolutional neural network starting from not only raw images. Furthermore, certain transformations may be applied to reduce variations that are not related to the problem at hand, for example, unwrapping a doughnut-shaped vessel cross-section into a rectangular representation with a normalized coordinate system before feeding it to the network. These front-end pipeline stages simultaneously mitigate both of the two drawbacks of using CNNs for medical imaging.

In general, early convolutional layers act as feature extractors that increase specificity, and one or two final fully connected layers act as classifiers (e.g., "softmax layers"). Schematic representations of the order of layers in a typical CNN and their functions are available from many sources.

Advantageously, the disclosed systems and methods utilize augmented datasets to enable non-invasive phenotyping of tissues assayed by radiological datasets. One type of augmentation is to pre-process the data to perform tissue classification and use "false color" overlays to provide an objectively verifiable dataset (as opposed to using only raw images, which does not have this possibility). Another type of augmentation is to use transformations on coordinate systems to highlight biologically feasible spatial context while removing noise variations to improve classification accuracy, allow smaller training sets, or both.

In an exemplary embodiment, the subject application systems and methods may employ the following multi-stage pipeline: (i) semantic segmentation to identify and classify regions of interest (e.g., that may represent quantitative biological analytes), (ii) spatial unwrapping to convert cross-sections of tubular structures (e.g., vein/artery cross-sections) into rectangles, and (iii) application of a trained CNN to read the annotated rectangles and identify which phenotypes (e.g., stable or unstable plaque and/or normal or abnormal FFR) are associated with them and/or predicted time to event (TTE). Note that by training and testing the CNN using the unwrapped dataset (with unwrapping) and the donut dataset (without unwrapping), it is demonstrated that unwrapping improves the validation accuracy of each particular embodiment. Thus, various implementations of imaging of tubular structures (e.g., plaque phenotyping) or other structures (e.g., lung cancer mass subtyping), or other applications, may similarly benefit from performing similar steps (e.g., semantic segmentation followed by spatial transformation such as unwrapping (before applying a CNN)). However, in some alternative embodiments, it is contemplated that an untransformed dataset (e.g., a spatially unwrapped dataset) may be used in determining the phenotype (e.g., in combination with or independent of the untransformed dataset).

In an exemplary embodiment, the semantic segmentation and spatial transformation may include the following: The image volume may be preprocessed, including target initialization, normalization, and any other desired preprocessing, such as deblurring or restoration, to form a region of interest that contains the physiological target to be phenotyped. In particular, the region of interest may be a volume constructed from cross sections through the volume. The body part may be determined automatically or explicitly provided by the user. Targets of body parts that are tubular in nature may be accompanied by a centerline. The centerline, if present, may branch. The branches may be labeled automatically or by the user. It is noted that a generalization regarding the concept of a centerline may be expressed for anatomical structures that are not tubular but benefit from some structural orientation, e.g., regions of tumors. In either case, a centroid may be determined for each cross section within the volume. For tubular structures, the centroid may be the center of a channel, e.g., the lumen of a blood vessel. For lesions, the centroid may be the centroid of the tumor. The (optionally deblurred or restored) images can be represented in an orthogonal dataset, where x represents the distance from the centroid, y represents the rotation theta, and z represents the cross-section. One such orthogonal set will be formed per branch or region. If multiple sets are used, "null" values may be used for overlapping regions, i.e., each physical voxel may be represented only once across sets in such a way that they fit geometrically with each other. Each dataset can be paired with an additional dataset with sub-regions labeled by objectively verifiable tissue composition. Examples of vascular tissue labels can include lumen, calcification, LRNC, IPH, etc. Examples of lesion labels can include necrotic, neovascularized, etc. These labels can be verified objectively, e.g., by histology. The paired datasets can be used as input to a training step to build a convolutional neural network. In an exemplary embodiment, two levels of analysis can be supported, the first at the individual cross-section level and the second at the volume level. The output labels represent phenotypes or risk stratification.

Exemplary image pre-processing may include, for example, deblurring or restoration using a patient-specific point spread determination algorithm to mitigate artifacts or image limitations resulting from the imaging process. These artifacts and image limitations may reduce the ability to determine characteristics predictive of phenotype. Deblurring or restoration may be achieved, for example, as a result of iteratively fitting a physical model of the scanner point spread function with normalized assumptions about the true underlying densities of different regions of the image.

In exemplary embodiments, the CNN may be an AlexNet, Inception, CaffeNet, or other network. In some embodiments, refactorings can be performed on the CNN, e.g., where the same number and types of layers are used but the dimensions of the inputs and outputs are changed (e.g., changing the aspect ratio). Exemplary implementations of various exemplary CNNs are provided as open source, e.g., in TensorFlow, and/or in other frameworks available as open source and/or licensed configurations.

In an exemplary embodiment, the dataset may be augmented. For example, in some embodiments, a 2D or 3D rotation may be applied to the dataset. Thus, for an untransformed (e.g., donut) dataset, augmentation may include, for example, randomly rotating the dataset (e.g., by a random angle between 0 and 360 degrees) as well as randomly flipping the dataset horizontally. Similarly, for a transformed (e.g., unwrapped) dataset, augmentation may include, for example, randomly flipping the image horizontally, such as by a random number of pixels ranging from 0 to the width of the image, in conjunction with a random "scrolling" of the image (where scrolling is analogous to rotating around theta).

In an exemplary embodiment, the dataset may be augmented by using different colors to represent different tissue analyte types. These colors may be selected to visually contrast with each other and with non-analyte surfaces (e.g., normal walls). In some embodiments, non-analyte surfaces may be depicted in gray. In an exemplary embodiment, the dataset augmentation may not only result in ground truth annotation of tissue features (e.g., tissue features indicative of a plaque phenotype), but also provide a spatial context of how such tissue features are present in a cross section (e.g., taken orthogonal to the axis of the vessel). Such spatial context may include a coordinate system (e.g., based on polar coordinates related to the centroid of the cross section) that provides a common basis for analysis of the dataset related to the histological cross section. Thus, the augmented dataset may be advantageously overlaid on top of the color-coded pathologist annotations (or vice versa). Advantageously, the histology-based annotated dataset may be used for training (e.g., training a CNN) in combination with or independent of image feature analysis of the radiology dataset. In particular, histology-based annotated datasets may improve the efficiency of DL approaches, since they use relatively simple false-color images instead of full high-resolution images, without losing spatial context. In an exemplary embodiment, coordinate directions may be represented internally using unit phasors and phasor angles. In some embodiments, the coordinate system may be normalized, for example, by normalizing the radial coordinates with respect to the wall thickness (e.g., to provide a common basis for comparing tubular structures/cross sections of different diameters/thicknesses). For example, the normalized radial distance has a value of 0 at the inside (inner wall lumen boundary) and a value of 1 at the outside boundary (outer wall boundary). In particular, this may be applied to tubular structures associated with blood vessels or other pathophysiology (e.g., the gastrointestinal tract).

Advantageously, the augmented dataset of the subject application provides in vivo non-invasive image-based classification based on known ground truth (e.g., a tissue classification scheme can be used to non-invasively determine phenotype). In some embodiments, the known ground truth may be non-radiological (such as histology or another ex vivo based tissue analysis). Thus, for example, a radiology dataset annotated to include ex vivo ground truth data (such as histology information) may be advantageously used as input data for a classifier. In some embodiments, multiple different known ground truths may be used in combination with each other or independently of each other in annotating the augmented dataset.

As described herein, in some embodiments, the augmented dataset may utilize a normalized coordinate system to avoid extraneous variations associated with, for example, wall thickness and radial presentation. Additionally, as described herein, in exemplary embodiments, the "donut" shaped dataset may be "unwrapped," for example, prior to classification training (e.g., using a CNN) and/or prior to running a training classifier on the dataset. Notably, in such embodiments, analyte annotation of the training dataset may be timed prior to transformation, e.g., after unwrapping, or a combination of both. For example, in some embodiments, the untransformed dataset may be annotated (e.g., using ex vivo classification data such as histology information) and then transformed for classifier training. In such embodiments, the finer granularity of ex vivo-based classification may be collapsed to match the lower intended granularity of in vivo radiological analysis, reducing computational complexity as well as simultaneously addressing what is otherwise subject to criticism of being a "black box."

In some embodiments, the colors and/or axes for visualizing the annotated radiological dataset may be selected to correspond to the same colors/axes typically shown in ex vivo ground truth based classifications (e.g., the same colors/axes used in histology). In an exemplary embodiment, the transformed augmented dataset (which may be normalized to wall thickness, for example) may be shown, where each analyte may be visually shown in a different contrasting color, against background regions (e.g., black or gray), against all non-analyte regions. In particular, depending on the embodiment, they may or may not be annotated to a common background, and thus may or may not visually distinguish between non-analyte regions inside and outside the vessel wall, or between background features (endoluminal surface irregularities, wall thickness variations, etc.). Thus, in some embodiments, the annotated analyte regions (e.g., color-coded and normalized to wall thickness) may be visually depicted against a uniform (e.g., completely black, completely gray, completely white, etc.) background. In other embodiments, the annotated analyte regions (e.g., color-coded but not normalized to wall thickness) may be visually depicted against an annotated background (e.g., using different shades (gray, black, and/or white) of (i) the central luminal region inside the lumen of the tubular structure, (ii) the non-analyte region inside the wall, and/or (iii) the outer region of the outer wall. This may allow analysis of variations in wall thickness (e.g., due to ulceration or thrombus). In further exemplary embodiments, the annotated dataset may include, for example, identification (and visualization) of regions of intraplaque hemorrhage and/or other morphological aspects. For example, regions of intraplaque hemorrhage may be visualized in red and LRNC in yellow.

One particular implementation of the subject systems and methods of application may be to direct vascular therapy. Classifications may be established according to the likely dynamic behavior of the plaque lesion (based on its physical properties or specific mechanisms, e.g., inflammatory or cholesterol metabolism) and/or based on disease progression (e.g., early vs. late in its natural history). Such classifications can be used to direct patient treatment. In an exemplary embodiment, the Stary plaque phenotyping system adopted by the AHA is utilized as an underlay, with the in vivo determined types shown in a color overlay. An exemplary mapping is "I", "II", "III", "IV", "V", "VI", "VII", "VIII"], resulting in class_map = [occult, occult, occult, occult, occult, unstable, unstable, stable, stable]. However, the systems and methods of the present disclosure are not tied to Stary. As another example, the Virmani system ["calcified nodule", "CTO", "FA", "FCP", "healed plaque rupture", "PIT", "IPH", "rupture", "TCFA", "ULC"] has been used with class_map = [stable, stable, stable, stable, stable, stable, unstable, unstable, unstable, unstable], and other typing systems may perform similarly well. In an exemplary embodiment, the disclosed system and method may merge heterogeneous typing systems, change the class map, or have other variations. In the case of FFR phenotypes, values such as normal or abnormal may be used, and/or numeric values may be used, for example to facilitate comparison with physical FFR.

Thus, in an exemplary embodiment, the disclosed systems and methods can provide plaque phenotypic classification based on an augmented radiological data set. In particular, the phenotypic classification(s) may include distinguishing stable from unstable plaques, for example where the ground truth basis of the classification is based on factors such as (i) luminal narrowing (which may be augmented by additional measurements such as tortuosity and/or ulceration), (ii) calcium content (which may be increased by depth, shape, and/or other complex presentations), (iii) lipid content (which may be increased by measurements of cap thickness and/or other complex presentations), (iv) anatomy or shape, and/or (v) IPH or other content. In particular, this classification has been demonstrated to have high overall accuracy, sensitivity, specificity, and a high degree of clinical relevance (which may change the existing standard of care for patients undergoing catheterization and cardiovascular care).

Another exemplary embodiment is lung cancer, where subtypes of tumor masses can be determined to direct the most beneficial treatment for patients based on the manifesting phenotype. In particular, pre-processing and dataset augmentation can be used to separate solid vs. semi-solid ("ground glass") sub-regions, which not only differ in malignancy but also suggest different optimal treatments.

In further exemplary embodiments, the systems and methods of the present disclosure may provide image pre-processing, image denoising, and novel geometric representations (e.g., affine transformations) of CT angiography (CTA) diagnostic images to facilitate and maximize the performance of best-in-class classifiers and CNN-based deep learning algorithms to develop markers of risk of adverse cardiovascular effects during the representation procedure. Thus, as disclosed herein, image deblurring or restoration may be used to identify lesions of interest and quantitatively extract plaque composition. Furthermore, a transformation of cross-sectional (along the main axis vessel) segmented images to, for example, an "unwrapped" rectangular reference frame following the lumen established along the X-axis may be applied to provide a normalized frame to enable the DL approach to best learn the representational features.

Note that while the exemplary embodiments herein utilize 2D annotated cross sections for analysis and phenotyping, application of the subject matter is not limited to such embodiments. In particular, some embodiments may utilize an augmented 3D dataset, e.g., instead of or in addition to processing 2D cross sections separately. Thus, in exemplary embodiments, video interpretation from computer vision may be applied to the classifier input dataset. Note that sequential processing of multiple cross sections, as in a "movie" sequence along the centerline, may generalize these methods for tubular structures, e.g., moving up and down the centerline, and/or other 3D manifestations depending on the aspect that best suits the anatomy.

In further exemplary embodiments, the false color representation in the augmented dataset may have a continuous value across pixel or voxel locations. This value may be used for "radiomics" features, with or without explicit validation, or for validated tissue types, and may be calculated independently for each voxel. Such a set of values may be present in any number of pre-processed overlays and fed to the phenotype classifier. In particular, in some embodiments, each pixel/voxel may have values for any number of different features (e.g., may be represented in any number of different overlays for different analytes, sometimes referred to as "multiple occupancy"). Alternatively, each pixel/voxel may be assigned to only one analyte (e.g., assigned to only a single analyte overlay). Furthermore, in some embodiments, the pixel/voxel value for a given analyte (e.g., a given analyte) may be based on an all-or-nothing classification scheme (e.g., the pixel is either calcium or not). Alternatively, the pixel/voxel value for a given analyte may be a relative value, e.g., a probability score. In some embodiments, the relative values of pixels/voxels are normalized across the set of analytes (e.g., so that the total probabilities sum to 100%).

In an exemplary embodiment, the classification model may be trained in whole or in part by application of multi-scale modeling techniques, such as partial differential equations, to represent, for example, likely cell signaling pathways or biologically motivated feasible representations.

Other alternative embodiments include using change data, e.g., collected from multiple time points, rather than (only) data from a single time point. For example, if the amount or quality of a negative cell type increases, it can be said to be a "progressor" phenotype, as opposed to a "regressor" phenotype if it decreases. A regressor may be due, for example, to a response to a drug. Alternatively, if the rate of change, e.g., of the LRNC, is fast, this may signify a different phenotype, e.g., a "rapid progressor."

In some embodiments, non-spatial information, such as that derived from other assays (e.g., laboratory results), demographics/risk factors, or other measurements taken from radiology images, may be fed into a final layer of the CNN to combine the spatial information with the non-spatial information.

Notably, although the systems and methods focus on phenotype classification, similar approaches can be applied with respect to outcome prediction. Such classification may be based on ground truth historical outcomes assigned to the training dataset. For example, life expectancy, quality of life, treatment efficacy (including comparison of different treatment methods), and other outcome predictions can be determined using the systems and methods of the subject application.

Examples of the subject application of the systems and methods are further illustrated in the drawings and detailed description below.

In further exemplary embodiments, the disclosed systems and methods provide for the determination of fractional fractional flow reserve in myocardial and/or brain tissue by measurement of plaque morphology. The disclosed systems and methods may use sophisticated methods to characterize the vasodilatory capacity of blood vessels through objectively verified determination of tissue types and characteristics that affect its distensibility. In particular, plaque morphology may be used as input to an analysis of the dynamic behavior of the vasculature in terms of fractional flow reserve (fractual fractional flow reserve data is used to train a model). Thus, it is possible to determine the dynamic behavior of the system, rather than a static description (only). Stenosis itself is well known for its poor predictive power, in that it only provides a static description, and the most accurate imaging-based assessment of dynamic function requires the addition of accurate plaque morphology. The disclosed systems and methods provide for determining accurate plaque morphology, which is then processed to determine dynamic function.

In an exemplary embodiment, deep learning is utilized to preserve the spatial context of tissue properties and vascular anatomy (collectively referred to as "plaque morphology") at an optimal level of granularity, while avoiding excessive non-material variation in the training set, as is required to go beyond other more simple uses of machine learning. Alternative methods by others use only measurements of vessel structure rather than a more complete treatment of tissue properties. Such methods may capture lesion length, stenosis, and possibly inlet and outlet angles, but ignore the determinants of vessel dilatation capacity. Use of these models requires high-level assumptions to be made regarding the flexibility of the entire arterial tree, but plaque and other tissue properties cause the distensibility of the coronary artery tree to distensibly in a non-uniform manner. Various parts of the tree are more or less distensible. As distensibility is a key factor in determining FFR, the methods fall short. Other methods that attempt to perform tissue characterization do so without objective validation of their accuracy and/or without the data augmentation methods necessary to preserve optimal spatial context for medical image deep learning (e.g., unwrapping and transformations such as verified false color tissue type overlays) necessary to provide efficacy for deep learning methods. Some methods attempt to increase training set size by using synthetic data, but this is ultimately limited to the limited data on which the synthetic is generated and is a data augmentation scheme rather than a true extension of the input training set. Additionally, the systems and methods of the present disclosure can create continuous assessments throughout the length of the vessel.

The disclosed system and method effectively leverages objective tissue characterization verified by histology across multiple arterial beds. Relevant to the exemplary application of arteriosclerosis, plaque composition is similar in coronary and carotid arteries, regardless of their age, which would largely determine the relative stability and suggest a similar presentation in CCTA as well as CTA. Minor differences in the extent of various plaque features include thicker caps in coronary arteries and a higher prevalence of intraplaque hemorrhage and calcified nodules in carotid arteries, but no difference in the nature of plaque components. In addition, carotid and coronary arteries have many similarities in the physiology of vascular tone regulation that impacts plaque evolution. Myocardial blood perfusion is regulated by epicardial coronary vasodilation in response to various stimuli, such as NO, which can cause dynamic changes in coronary tone and result in multiple changes in blood flow. Similarly, carotid arteries are not just conduits supporting cerebral circulation, they demonstrate vasoreactive behavior in response to stimuli including changes in shear stress. Endothelial shear stress contributes to endothelial health and favorable vessel wall transcriptome profiles. Clinical studies have demonstrated that regions of low endothelial shear stress are associated with the development of atherosclerosis and high-risk plaque features. Similarly, in the carotid arteries, lower wall shear stress has been associated with plaque development and localization. (Endothelial shear stress is a useful measurement in its own right, but does not replace plaque morphology.) Technical challenges vary by bed (e.g., use of gating, vessel size, amount and nature of motion), but these effects are mitigated by the scanning protocol, resulting in approximate in-plane voxel sizes in the range of 0.5-0.75 mm, and it is important to recognize that the in-plane resolution of coronary arteries (smaller vessels) is actually superior to that of carotid arteries (voxels are isotropic in coronary arteries, but not in the neck and peripheral limbs).

The present disclosure is based on sound mathematical principles that respect the Nyquist-Shannon sampling theorem to achieve effective resolution in routinely acquired CTA in much the same way as IVUS VH. IVUS imaging has excellent spatial resolution for the overall structure (i.e., lumen), but generally lacks the ability to characterize plaque components with high precision. In the literature, IVUS resolution is estimated at 70-200 μm axially and 200-400 μm laterally, using typical transducers in the 20-40 MHz range. IVUS VH is a method of spectral backscatter analysis that allows plaque compositional analysis (and therefore measurement). For IVUS VH techniques that use a large axial (e.g., 480 μm) moving window, the relatively large size of this moving window (and therefore the precision of the compositional analysis) is fundamentally limited by the bandwidth requirements of the spectral analysis. If the IVUS VH images are viewed in a small moving window, for example 250 μm, each IVUS pixel will be classified into a separate category, limiting the accuracy of this analysis. 64-slice multi-detector CCTA scans have been described as ranging in resolution from 300 to 400 μm. This already brings the CCTA resolution very close to that of IVUS VH, but there are additional factors specific to the present analysis that must be considered. Thus, rather than classifying CCTA pixels discretely, the disclosed systems and methods perform an iterative deblurring or restoration modeling step with sub-voxel accuracy, for example using a triangular tessellated surface to represent the true surface of the lipid core. The lipid core area ranges from 6.5 to 14.3 mm2 in 393 patients (with a corresponding radius of curvature of 1.4 to 2.1 mm). Using the formula for the chord length where the chord spans a single voxel diagonally, this represents an upper limit on the error of the tessellated surface representation of the lipid core at 44 μm. For a total accuracy range of 194-244 μm, which is typically comparable to the accuracy of IVUS VH for measuring cap thickness, there are additional factors associated with the deblurring or restoration analysis that can introduce errors on the order of half a pixel.

The present disclosure is also innovative in addressing the fundamental limitations of the application of artificial intelligence and deep learning to the analysis of arteriosclerosis imaging data. Conventional competitive approaches, lacking a validated objective foundation such as , are fundamentally limited in multiple ways. First, arbitrary thresholds are used, and therefore accuracy cannot be assessed except through weak forms of correlation with other markers, which themselves lack objective validation. Second, this lack of objectivity increases the demand for large amounts of data on which to base correlations in context. This places unfeasible demands on the manual annotation of radiological images, which are themselves the subject of analysis (i.e., as opposed to being validated from an independent modality). Third, due to the interpretability of the resulting models, these models must be presented as a "black box" to regulatory agencies such as the FDA, lacking a scientifically rigorous elucidation of the mechanism of action that can be coupled with traditional biological hypothesis testing. Specifically, CNNs have proven to be good at performing many different computer vision tasks, but when applied to radiology datasets, they have the following major drawbacks: 1) they require large training and validation datasets; and 2) intermediate CNN calculations do not generally represent measurable properties, making regulatory approval difficult. To address these challenges, we utilize a pipeline approach consisting of stages with outputs that can be individually objectively validated at the biological level and fed to the CNN. The present invention overcomes these drawbacks by using a pipeline consisting of one or more stages that are biologically measurable (i.e., can be objectively validated), followed by a smaller range of convolutional neural network processing on these validated biological properties, outputting desired output conditions rather than based on subjective or qualitative "imaging features". These architectural features mitigate the drawbacks by increasing the efficiency of the available training data, allowing the intermediate steps to be objectively validated. The disclosed system and method simultaneously mitigates the drawbacks of using CNNs in medical imaging by 1) reducing the complexity of the vision task to an acceptable level when training a CNN on a moderately sized dataset, and 2) generating intermediate outputs that are objectively verified and easily interpretable by users or regulatory agencies. The intermediate CNN computations generally do not represent any measurable properties, spatial context is often difficult to obtain in ML approaches using feature extraction, and using raw datasets that include spatial context often lacks objective ground truth labels for the extracted features, both in traditional machine learning and deep learning approaches. Similarly, raw datasets contain a lot of variation that is "noisy" with respect to the classification problem at hand, which is overcome in non-medical computer vision applications by having very large training sets, especially sets annotated with ground truth, on a scale that is generally unavailable in medical applications.

The quantitative capabilities of the disclosed systems and methods make them ideal for the analysis of more advanced imaging protocols, such as early/delayed phase contrast, dual energy, and multispectral techniques being investigated for tissue characterization.

While the systems and methods of the present disclosure have been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the scope of the present disclosure.

The foregoing will be apparent from the following more particular description of exemplary embodiments, as illustrated in the accompanying drawings, in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments of the present disclosure.

1 depicts a schematic diagram of an exemplary system for determining and characterizing a medical condition by implementing a hierarchical analysis framework, in accordance with the present disclosure. We outline a resampling-based model building approach according to the present disclosure that can be implemented by the systems and methods described herein. 1 depicts a sample patient report according to the present disclosure that may be output by the systems and methods described herein. 1 depicts a sample patient report according to the present disclosure that may be output by the systems and methods described herein. 1 depicts a sample patient report according to the present disclosure that may be output by the systems and methods described herein. 1 depicts a sample patient report according to the present disclosure that may be output by the systems and methods described herein. 1 depicts exemplary segmentation levels of a multi-scale vessel wall analyte map according to the present disclosure. 1 depicts an exemplary pixel-level probability mass function as a set of analyte probability vectors in accordance with the present disclosure. 1 illustrates a technique for computing putative analyte blobs according to the present disclosure. 1 depicts normalized vessel wall coordinates of an exemplary vessel wall composition model in accordance with the present disclosure. 1 depicts an exemplary margin between plaque and the outer vessel wall that was removed for a histological specimen according to the present disclosure. 1 illustrates several complex vascular topologies that can be described using the techniques described herein, in accordance with the present disclosure. 1 depicts a representation of an exemplary analyte blob with a distribution of normalized vessel wall coordinates in accordance with the present disclosure. 1 depicts an example distribution of blog descriptors according to the present disclosure. 1 illustrates an exemplary model of imaging data correlating between hidden ground truth states and observed states in accordance with the present disclosure. 1 illustrates an example diagram of a Markov model/Viterbi algorithm for relating observed states to hidden states in an image model in accordance with the present disclosure. 1 depicts an exemplary frequency distribution of total number of blobs per histological slide for multiple histological slides, according to the present disclosure. 1 illustrates an exemplary transplantation of a 1D Markov chain in accordance with the present disclosure. 1 illustrates an exemplary first-order Markov chain example of a text probability table in accordance with the present disclosure. 1 depicts the conditional dependency of a first pixel based on its neighboring pixels, in accordance with the present disclosure. 1 depicts a further exemplary hierarchical analysis framework in accordance with the present disclosure. 1 illustrates an exemplary application of phenotyping for purposes of directing vascular therapy in accordance with the present disclosure. In the illustrated example, the Stary plaque phenotyping system adopted by the AHA is used as an underlay, with the in vivo determined type shown as a color overlay. 1 depicts an exemplary application of the present disclosure to phenotyping of lung cancer. 1 illustrates an exemplary image pre-processing step of deblurring or restoration according to the present disclosure. Deblurring or restoration mitigates artifacts or image limitations resulting from the imaging process that may reduce the ability to determine features predictive of phenotype using patient-specific point spread determination algorithms. The depicted deblurred or restored (processed) image (bottom) is derived from CT imaging of a plaque (top) and is the result of iteratively fitting a physical model of the scanner point spread function with normalizing assumptions about the true latent density of different regions of the image. 1 depicts an exemplary application demonstrating an enriched dataset of atherosclerotic plaque phenotypes in accordance with the present disclosure. Shown are tangential and radial variables that are internally represented using unit phasors and phasor angles (shown grayscale coded), illustrating the use of normalized axes for tubular structures relevant to blood vessels and other pathophysiology associated with such structures (e.g., the gastrointestinal tract). 1 shows an exemplary overlay of a radiology application of CTA-generated annotations (open areas) on top of pathologist-generated annotations from histology (solid areas) in accordance with the present disclosure. We demonstrate a further step of data augmentation according to the present disclosure, specifically utilizing a normalized coordinate system to avoid extraneous variations associated with wall thickness and radial presentation. Specifically, the "donut" is "unwrapped" while preserving the pathologist's annotations. 1 represents a further step in data augmentation relevant to plaque phenotyping according to the present disclosure. Working from the unwrapped form, luminal irregularities (e.g., from ulceration or thrombus) and locally varying wall thickness are represented. 1 depicts data-enhanced imaging (in both wrapped and unwrapped forms), including intraplaque hemorrhage and/or other morphologies, in accordance with the present disclosure. 1 depicts validation results of an algorithm trained and validated on different variations of data augmented imaging according to the present disclosure. 1 depicts validation results of an algorithm trained and validated on different variations of data augmented imaging according to the present disclosure. 1 depicts validation results of an algorithm trained and validated on different variations of data augmented imaging according to the present disclosure. An exemplary application of phenotyping lung lesions according to the present disclosure is provided. Examples of biological properties (e.g., including tissue characteristics, morphology, etc.) for phenotyping of lung lesions in accordance with the present disclosure are provided. Note that in an exemplary embodiment, false color may be represented as continuous values rather than discrete values for one or more of these biological properties. Examples of biological properties (e.g., including tissue characteristics, morphology, etc.) for phenotyping of lung lesions in accordance with the present disclosure are provided. Note that in an exemplary embodiment, false color may be represented as continuous values rather than discrete values for one or more of these biological properties. 1 depicts a high-level diagram of an example approach to user interaction with a computer-aided phenotyping system in accordance with the present disclosure. 1 illustrates an example system architecture according to the present disclosure. 1 illustrates components of an exemplary system architecture according to the present disclosure. 1 is a block diagram of an exemplary data analysis according to the present disclosure. Patient images are collected and the raw slice data is used with a set of algorithms to measure biological attributes that can be objectively verified, which are then formed into an augmented dataset to feed one or more CNNs, in this example the results are propagated forward and backward using a recursive CNN to implement constraints or create continuous conditions (e.g., monotonically decreasing fractional flow reserve from proximal to distal throughout the vascular tree, or constant HRP values at focal lesions, or other constraints, etc.). 1 illustrates the causation and available diagnosis of coronary ischemia according to the present disclosure. 1 depicts an exemplary 3D segmentation of the lumen, vessel wall, and plaque components (LRNC and calcification) in accordance with the present disclosure for two patients presenting with chest pain and similar risk factors and degree of stenosis. Left: 68-year-old male with NSTEMI at follow-up. Right: 65-year-old male with no events at follow-up. The systems and methods of the present disclosure successfully predicted the outcomes of each. 1 depicts an exemplary 3D segmentation of the lumen, vessel wall, and plaque components (LRNC and calcification) in accordance with the present disclosure for two patients presenting with chest pain and similar risk factors and degree of stenosis. Left: 68-year-old male with NSTEMI at follow-up. Right: 65-year-old male with no events at follow-up. The systems and methods of the present disclosure successfully predicted the outcomes of each. 1 depicts examples of histological processing steps for objective verification of tissue composition according to the present disclosure. 1 is a schematic depiction of multiple cross sections that may be processed to provide dynamic analysis across a scanned vascular tree in accordance with the present disclosure, showing the relationships between the cross sections, where the processing of one cross section is dependent on the processing of its adjacent cross sections;

Presented herein are systems and methods for analyzing lesions utilizing quantitative imaging. The disclosed systems and methods advantageously utilize a hierarchical analytical framework that identifies and quantifies biological properties/analytes from imaging data and then identifies and characterizes one or more lesions based on the quantified biological properties/analytes. This hierarchical approach, which examines the underlying biology using imaging as an intermediate to evaluate lesions, offers numerous analytical and processing advantages over systems and methods configured to directly determine and characterize lesions from raw imaging data without a validation step and/or without the advantageous processing described herein.

One advantage is the ability to utilize training sets from tissue sample sources, such as, for example, histological information, from non-radioactive sources, in combination with or independent of the training set of radioactive sources, to correlate radiological imaging features with biological properties/analytes and lesions. For example, in some embodiments, the histological information may be used in a training algorithm to identify and characterize one or more lesions based on the quantified biological properties/analytes. More specifically, biological properties/analytes identifiable/quantifiable in non-radioactive data (e.g., in an invasively acquired histological dataset or obtainable via gene expression profiling) may also be identified and quantified in the radiological data (which is advantageously non-invasive). These biological properties/analytes can then be correlated with clinical findings regarding lesions using information from non-radioactive sources, for example, utilizing histological information, gene expression profiling, or other clinically rich datasets. This set of clinically correlating data can then serve as a training set or part of a training set to determine/tune (e.g., utilizing machine learning) algorithms that correlate biological properties/analytes with lesions that have a known relationship to clinical outcomes. These algorithms that correlate biological properties/analytes with lesions derived utilizing the non-radiation source training set can then be applied to the evaluation of biological properties/analytes derived from the radiological data. Thus, the systems and methods of the present disclosure may advantageously enable the use of radiological imaging (which may advantageously be cost-effective and non-invasive) to provide a surrogate means for predicting clinical outcomes or guiding treatment.

In particular, in some cases, training data from non-radioactive sources (such as histological information) may be more accurate/reliable than training data from radioactive sources. Furthermore, in some embodiments, training data from non-radioactive sources may be used to augment training data from radioactive sources. Thus, the disclosed hierarchical analysis framework advantageously improves the trainability and resulting reliability of the algorithms disclosed herein, since better data input is likely to result in better data output. As noted above, one important advantage is that once trained, the systems and methods of the present disclosure may enable the derivation of clinical information comparable to existing histological and other non-radioactive diagnostic types of tests, without the need to undergo invasive and/or costly procedures.

Alternatively, in some embodiments, training sets for non-radiographic sources (such as non-radiographic imaging sources, e.g., histological sources, and/or non-imaging sources) may be utilized in combination with or independent of training sets for radiological sources, e.g., in correlating image features to biological properties/analytes. For example, in some embodiments, one or more biological models may be extrapolated and fitted to correlate the radiological and non-radiographic data. For example, histological information may be correlated with radiological information based on an underlying biological model. This correlation may advantageously enable training of recognition of biological properties/analytes in radiological data utilizing non-radiographic, e.g., histological, information.

In some embodiments, data derived from complementary modalities can be used, for example, to correlate image features with biological properties/analytes from blood panels, physical FFR, and/or other data sources.

In an exemplary embodiment, one or more biological models may be extrapolated and fitted utilizing imaging data derived from one imaging modality correlated and/or fused with another imaging modality or a non-imaging source such as a blood test. These biological models may advantageously correlate between the imaging and non-imaging data sets based on the biological models. Thus, these biological models may enable the hierarchical analysis framework to utilize data from one imaging modality with another imaging modality or non-imaging sources in identifying/quantifying one or more biological properties/analytes or identifying/characterizing one or more disease states.

Another advantage of the hierarchical analysis framework disclosed herein is that data from multiple same or different types of data sources can be incorporated into the process of identifying and characterizing a lesion based on imaging data. For example, in some embodiments, one or more non-imaging data sources can be used in combination with one or more imaging data sources in identifying and quantifying the set of biological properties/analytes. Thus, among other things, the set of biological properties/analytes may include one or more biological properties/analytes identified and/or quantified based on one or more imaging data sources, one or more biological properties/analytes identified and/or quantified based on one or more non-imaging data sources, and/or one or more biological properties/analytes identified and/or quantified based on a combination of imaging and non-imaging data sources (note that for purposes of the quantitative imaging systems and methods of the present disclosure, the set of biological properties/analytes may generally include at least one or more biological properties/analytes identified and/or quantified based on at least a portion of the imaging data). The ability to extend information from imaging data sources with information from other imaging and/or non-imaging data sources in identifying and quantifying a set of biological properties/analytes adds to the robustness of the systems and methods presented herein, allowing for the utilization of any and all relevant information in identifying and characterizing lesions.

Yet another advantage of the hierarchical analytical framework includes the ability to adjust/fine-tune the data at each level, e.g., before or after utilizing that data to evaluate a later level (note that in some embodiments this may be an iterative process). For example, in some embodiments, information related to a set of identified and quantified biological properties/analytes may be adjusted a posteriori (e.g., after their initial identification and/or quantification). Similarly, in some embodiments, information related to a set of identified and characterized lesions may be adjusted a posteriori (e.g., after their initial identification and/or characterization). These adjustments may be automatic or user-based, and may be objective or subjective. The ability to adjust/fine-tune the data at each level may advantageously improve the accountability and reliability of the data.

In an exemplary embodiment, the adjustment can be based on context information that can be used to update one or more probabilities that affect the determination or quantification of a biological property/analyte. In an exemplary embodiment, the context information for post-hoc adjustment of information related to the set of identified and quantified biological properties/analytes may include patient demographic information, correlations between biological properties/analytes, or correlations between identified/characterized lesions and biological properties/analytes. For example, in some cases, the biological properties/analytes may be related in the sense that the identification/quantification of a first biological property/analyte may affect the probability associated with the identification/quantification of a second biological property/analyte. In other cases, for example, the identification/characterization of a first lesion based on an initial set of identified/quantified biological properties/analytes may affect the probability associated with the identification/quantification of a biological property/analyte that is in the initial set, or even a biological property/analyte that was not in the first set. In further cases, the lesions may be related, for example, the identification/characterization of a first lesion may affect the probability associated with the identification/characterization of the first lesion. As described above, information related to biological property/analyte identification and quantification and/or pathology identification and characterization may be iteratively updated, for example, until data convergence or a threshold/benchmark is achieved or for a selected number of cycles.

Further advantages of the hierarchical analysis framework include the ability to provide a user, e.g., a physician, with information related to both the lesion and the underlying biology. This added context can facilitate not only the clinical diagnosis/assessment, but also the evaluation/decision of next steps, e.g., therapy/treatment options or further diagnoses. For example, the systems and methods may be configured to determine which biological parameters/analytes related to the identification/quantification of one or more lesions are most uncertain/have the highest degree of uncertainty (e.g., due to lack of data or conflicting data). In such cases, a particular further diagnosis may be recommended. The additional context of providing a user with information related to both the lesion and the underlying biology may further help the user evaluate/error check various clinical conclusions and recommendations reached by the analysis.

As used herein, a hierarchical analysis framework refers to an analysis framework in which one or more intermediate sets are utilized as intermediate processing layers or intermediate transformations between an initial set of data points and a final set of data points. This framework is similar to the concept of deep learning or hierarchical learning, which uses algorithms to model higher levels of abstraction using multiple processing layers or otherwise utilizing multiple transformations, such as multiple nonlinear transformations. In general, the hierarchical analysis framework of the disclosed systems and methods includes data points related to biological properties/analytes as intermediate processing layers or intermediate transformations between imaging data points and lesion data points, and in exemplary embodiments may include multiple processing layers or multiple transformations (e.g., as embodied by multiple levels of data points) to determine each of the imaging information, the underlying biological information, and the pathological information. Although an exemplary hierarchical analysis framework structure (e.g., having specific processing layers, transformations, and data points) is introduced herein, the disclosed systems and methods are not limited to such implementations. Rather, any number of different types of analysis framework structures may be utilized without departing from the scope and spirit of the present disclosure.

In an exemplary embodiment, the hierarchical analysis framework of the subject application may be conceptualized as including a logical data layer as an intermediary between the empirical data layer (including imaging data) and the outcome layer (including pathology information). While the empirical data layer represents the data directly fed to it, the logical data layer advantageously adds a degree of logic and reasoning that distills this raw data into a set of useful analytes for the outcome layer in question. Thus, for example, empirical information from a diagnosis, such as raw imaging information, may be advantageously distilled into logical information related to a particular set of biological features that are relevant for evaluating a selected lesion or group of lesions (e.g., lesions associated with an imaged region of a patient's body). In this manner, the biological features/analytes of the subject application may also be thought of as pathological symptoms/indicators.

The biological features/analytes of the subject application may be referred to herein as biomarkers. Although the term "biological" or the prefix "bio" is used to characterize a biological feature or biomarker, this is intended only to mean that the feature or marker has some degree of relevance with respect to the patient's body. For example, a biological feature may be anatomical, morphological, compositional, functional, chemical, biochemical, physiological, histological, genetic, or any number of other types of features associated with the patient's body. By way of example, biological features (e.g., associated with a particular anatomical region of a patient, such as the vascular system, respiratory system, organs such as the lungs, heart, or kidneys, or other anatomical regions) utilized by certain embodiments of the presently disclosed systems and methods are disclosed herein.

Although the exemplary systems and methods of the present disclosure may be directed to the detection, characterization, and treatment of a lesion/disease, the application of the systems and methods of the present disclosure is not limited to lesions/diseases, but rather may be more generally applicable with respect to any clinically relevant medical condition of a patient, including, for example, syndromes, disorders, trauma, allergic reactions, etc.

In exemplary embodiments, the systems and methods of the present disclosure relate to computer-assisted phenotyping, e.g., using knowledge of biology to analyze medical images to indicate a phenotype that measures differences between disease types determined through research and thus predicts outcome. Thus, in some embodiments, characterizing a lesion may include determining a phenotype of the lesion, which in turn can determine a predicted outcome.

With initial reference to FIG. 1, a schematic diagram of an exemplary system 100 is depicted. There are three basic functions that may be provided by the system 100, as represented by a trainer module 110, an analyzer module 120, and a cohort tool module 130. As depicted, the analyzer module 120 advantageously implements a hierarchical analysis framework that first utilizes a combination of (i) imaging features 122 from one or more acquired images 121A of the patient 50 and (ii) non-imaging input data 121B for the patient 50 to identify and quantify biological properties/analytes 130, and then identifies and characterizes one or more lesions (e.g., prognostic phenotypes) 124 based on the quantified biological properties/analytes 123. The analyzer module 120 can advantageously operate independently of ground truth or validation references by implementing one or more pre-trained, e.g., machine learning algorithms to draw its inferences.

In an exemplary embodiment, the analyzer may include algorithms for calculating imaging features 122 from acquired images 121A of the patient 50. Some of the image features 122 may be advantageously calculated on a voxel-by-voxel basis, while other image features may be calculated on a region-of-interest basis. Examples of non-imaging inputs 121B that may be utilized with the acquired images 121A may include data from laboratory systems, patient-reported symptoms, or the patient's medical history.

As noted above, the image features 122 and non-imaging inputs may be utilized by the analyzer module 120 to calculate biological properties/analytes 123. In particular, biological properties/analytes are typically quantitative objective properties (e.g., objectively verifiable, rather than described as an impression or appearance) that may represent, for example, the presence and degree of a marker (e.g., a chemical substance) or other measurements such as the structure, size, or anatomical properties of an area of interest. In an exemplary embodiment, the quantified biological properties/analytes 123 may be displayed or exported for direct utilization by a user, e.g., a clinician, in addition to or independent of further processing by the analyzer module.

In an exemplary embodiment, one or more of the quantified biological properties/analytes 123 can be used as inputs to determine a phenotype. Phenotypes are typically defined in a disease-specific manner independent of imaging and are often derived from ex vivo pathophysiological samples where a relationship to an expected outcome has been documented. In an exemplary embodiment, the analyzer module 120 can also provide a predicted outcome 125 for the determined phenotype.

It should be understood that exemplary implementations of the analyzer module 120 are further described herein with respect to specific embodiments that follow the general description of the system 100. In particular, specific imaging features, biological properties/analytes, and lesions/phenotypes are described with respect to specific medical applications, such as the vascular or respiratory systems.

With further reference to FIG. 1, the cohort tools module 130 allows for defining a cohort of patients for group analysis of patients, for example, based on a selected set of criteria relevant to the cohort study in question. An exemplary cohort analysis may be for a group of patients enrolled in a clinical trial, where, for example, the patients may be further grouped based on one or more arms of the trial, such as treatment versus control. Another type of cohort analysis may be for a set of subjects for which a ground truth or reference exists, where this type of cohort may be further decomposed into a training or "development" set and a test or "holdout" set. A development set may be supported for training algorithms and models in the analyzer module 120 (112), and a holdout set may be supported for evaluating/validating the performance of algorithms or models in the analyzer module 120 (113).

Continuing with reference to FIG. 1, the trainer module 110 may be utilized to train 112 algorithms and models in the analyzer module 120. In particular, the trainer module 110 may rely on ground truth 111 and/or reference annotations 114 to derive weights or models, for example, according to established machine learning paradigms or by informing the algorithm developer. In an exemplary embodiment, classification and regression models are used, which may be highly adaptable and capable of uncovering complex relationships between predictors and responses, for example. However, the ability of these models to adapt to underlying structures in existing data may enable the models to find patterns that cannot be reproduced in another sample of subjects. Adapting to irreproducible structures in existing data is commonly known as overfitting the model. To avoid building overfitting models, a systematic approach may be applied that prevents the final model from finding spurious structures and allows the end user to have confidence that the model will predict new samples with a similar degree of accuracy on the set of data on which it was evaluated.

Successive training sets can be used to determine optimal tuning parameter(s) and test sets can be used to estimate predictive performance of the algorithm or model. The training sets can be used to train each of the classifiers via randomized cross-validation. The dataset can be iteratively split into training and test sets and used to determine classification performance and model parameters. Splitting the dataset into training and test sets is done using a stratified or maximum dissimilarity approach. In an exemplary embodiment, a resampling approach (e.g., bootstrap) can be used within the training set to obtain (i) optimal parameter estimates and (ii) confidence intervals for the predictive performance of the model.

Figure 2 shows an overview of a resampling-based model building approach 200 that may be utilized by the systems and methods of the present disclosure. First, in step 210, a set of tuning parameters may be defined. Then, in step 220, for each set of tuning parameters, the data is resampled, a model is fitted, and holdout samples are predicted. In step 230, the resampling estimates are combined into a performance profile. Then, in step 240, the final tuning parameters may be determined. Finally, in step 250, the final tuning parameters are refitted to the entire training set. After each model is tuned from the training set, each may be evaluated for predictive performance on the test set. The test set evaluation occurs once for each model to ensure that the model building process does not overfit the test set. For each model built, the best tuning parameter estimates, the performance on the resampled training set, and the performance on the test set may be reported. Then, the values of the model parameters for the randomized splits are compared to evaluate the stability of the model and its robustness to the training data.

According to the systems and methods of the present disclosure, several models may be tuned for each biological property/analyte (e.g., tissue type) represented in the ground truth map. Model responses may include, for example, covariance-based techniques, non-covariance-based techniques, and tree-based models. Depending on their structure, the endpoints may have continuous and categorical responses, with some of the above categorical techniques being used for both categorical and continuous responses, and other techniques being specific to either categorical or continuous responses. For each model, estimates of the optimal tuning parameters, performance on the resampled training set, and performance on the test set may be reported.

Table 1 above provides a summary of some of the exemplary functions of the analyzer module 120 of the system 100. That is, the analyzer module 120 may be configured to border fields, e.g., align multiple data streams across fields, segment organs, vessels, lesions, and other application-specific objects, and/or reformat/reconstruct anatomical structures for a particular analysis. The analyzer module 120 may be configured to border targets, e.g., lesions, within the bordered fields. Target bordering may include, e.g., aligning multiple data streams at a locale, performing fine-grained segmentation, measuring size and/or other characteristics of relevant anatomical structures, and/or extracting features of the entire target (e.g., biological properties/analytes characteristic of the entire target region). In some embodiments, one or more sub-target regions may also be bordered, e.g., the target region may be divided into sub-targets according to a particular application using sub-target-specific calculations (e.g., biological properties/analytes characteristics of the sub-target region). The analyzer module 120 may also, for example, border components or related features (e.g., composition) in a particular field, target, or sub-target region. This may include segmenting or re-segmenting the components/features, calculating values of the segmented components/features (e.g., biological properties/analyte properties of the components/features), and assigning a probability map to the readings. Subsequent lesions may be characterized, for example, by determining a phenotype and/or predicted outcome of the lesion based on the biologically quantified properties/analytes. In some embodiments, the analyzer module 120 may be configured to compare data across multiple time points, for example, one or more of the biological components/analytes may include time-based quantification. In further embodiments, a wide scan field may be utilized to evaluate multi-pathological lesions, for example, based on aggregate quantification of biological properties/analytes across multiple targets within the bordered field. Finally, based on the aforementioned analysis, the analyzer module 120 may be configured to generate a patient report.

A sample patient report 300 is shown in FIG. 3. As shown, the sample patient report 300 may include quantification of biological parameters/analytes, such as those related to structure 310 and composition 320, and data from non-imaging sources, such as hemodynamics 330. The sample patient report may further include visualizations 340, e.g., 2D and/or 3D visualizations of image data, as well as combined visualizations of non-imaging data, such as hemodynamic data overlaid on the imaging data. Various analyses 350 may be displayed to evaluate the biological parameters/analytes, including visualizations of one or more model(s) (e.g., decision tree models) for determining/characterizing the lesion. Patient background and identification information may also be included. Thus, the analyzer module 120 of the system 100 may advantageously provide a user, e.g., a clinician, with comprehensive feedback for evaluating the patient.

The systems and methods of the present disclosure can be advantageously adapted to specific applications. Exemplary vascular and pulmonary applications are described in more detail in the following sections (although it will be understood that the specific applications described have general meaning and interoperability with respect to many other applications). Table 2 provides an overview of vascular and pulmonary related applications utilizing the hierarchical analysis framework described herein.

The following sections provide specific examples of quantitative biological properties/analytes that can be utilized by the systems and methods of the present disclosure for vascular applications.
Anatomy: Vascular structure measurements, especially those leading to the determination of stenosis (%), have long been and continue to be the single most used measurements in patient care. These measurements were initially limited to measurements of the inner lumen, rather than wall measurements that include both the inner and outer surfaces of the vessel wall. However, unlike x-ray angiography, all major non-invasive modalities are able to resolve the vessel wall, which allows for extended measurements that may be achieved. As the category is broad and measurements are of objects of various sizes, generalizations must be made with caution. The main consideration is the spatial sampling or resolution limitations. However, minimally detectable changes in wall thickness may be lower than spatial sampling by taking advantage of small changes in intensity levels due to partial volume effects. Additionally, the resolutions listed generally refer to the grid size and field of view of the post-acquisition reconstruction, rather than the actual resolution of the imaging protocol, which determines the smallest feature size that can be resolved. Similarly, in-plane and through-plane resolution may or may not be the same, and not only the size of a given feature but also its nature and shape will determine the measurement accuracy. Last but not least, in some cases, formulaic conclusions are drawn from applying thresholds to measurements, which may then be interpreted according to signal detection theory with the ability to optimize the trade-off between sensitivity and specificity.

Tissue Features: Quantitative assessment of individual components of atherosclerotic plaque, including, for example, lipid-rich necrotic core (LRNC), fibrosis, intraplaque hemorrhage (IPH), permeability, and calcification, can provide important information regarding the relative structural integrity of the plaque, which can assist the physician in making decisions regarding a course of medical or surgical therapy. From an imaging technology perspective, the ability to do this depends less on spatial resolution than on contrast resolution and tissue differentiation, made possible by different tissues responding differently to the incident energy such that they produce different received signals. Each imaging modality has done this to some degree: ultrasound terms such as "echolucency," CT number in Hounsfield units, and differential MR intensity as a function of various sequences such as T1, T2, and T2* (but are not limited to these).

Dynamic tissue behavior (e.g., permeability): In addition to morphological features of the vessel wall/plaque, there is increasing recognition that dynamic features are valuable quantitative indicators of vascular pathology. Dynamic sequences, in which acquisitions are performed at multiple closely spaced intervals (known as phases), extend the repertoire beyond spatially resolved values of t to include temporally resolved values that can be used for compartmental modeling or other techniques to determine the dynamic response of tissue to stimuli (e.g., but not limited to, contrast wash-in and wash-out). Through the use of dynamic contrast-enhanced imaging using ultrasound or MR in the carotid artery or delayed contrast enhancement in the coronary arteries, sensitive assessments of the relative permeability (e.g., Ktrans and Vp parameters from mechanical analysis) of the neovascularized microvascular network within the plaque of interest can be determined. In addition, these dynamic series also help distinguish between increased vascular permeability and plaque infiltration and blood.

Hemodynamics: The fundamental hemodynamic parameters of the circulation have a direct impact on vascular disorders. Blood pressure, blood flow velocity, and vascular wall shear stress can be measured with techniques ranging from very simple oscillometry to advanced imaging analysis. Using general principles of fluid mechanics, calculations of vascular wall shear stress can be confirmed at various regions of the wall. Similarly, MRI, with or without the combination of US, has been used to calculate wall shear stress (WSS) and correlate the results with structural changes in the vessel of interest. In addition, the effects of antihypertensive drugs on hemodynamics have been followed up in short-term and long-term studies.

Thus, in an exemplary embodiment, an important aspect of applying the systems and methods of the present disclosure in a vascular setting may include assessing plaque structure and plaque composition. Assessing plaque structure may advantageously include, for example, lumen measurements (which improve stenosis measurements by providing area rather than just diameter measurements), and wall measurements (e.g., wall thickness and vascular remodeling). Assessing plaque composition may advantageously include quantification of tissue properties (e.g., lipid core, fibrosis, calcification, permeability, etc.) rather than merely designating "soft" or "hard" as typically found in the prior art. Tables 3 and 4 below set forth exemplary structure and tissue property calculations, respectively, that may be utilized by vascular applications of the systems and methods of the present disclosure.

An exemplary system relating to the assessment of the vasculature may advantageously include/use algorithms for assessing vascular structure. Thus, the system may, for example, use a target/vessel segment/cross-section model for segmenting the underlying structure of the imaged vessel. A fast-progressing competitive filter may advantageously be applied to separate vessel segments. The system may further be configured to process vessel bifurcations. Image registration may be applied utilizing Mattes mutual information (MR) or mean squared error (CT) metrics, rigid Versol transform, or LBFGSB optimizer, etc. As described herein, vessel segmentation may advantageously include lumen segmentation. Initial lumen segmentation may utilize trust connection filters (e.g., carotid, vertebral, femoral, etc.) to differentiate lumens. Lumen segmentation may utilize MR imaging (such as a combination of normalized, e.g., inverted images for dark contrast), or CT imaging (such as the use of registered pre-contrast, post-contrast CT, and 2D Gaussian distributions) to define vascular features. Various connected components can be analyzed and thresholds applied. Vessel segmentation may further involve outer wall segmentation (e.g., utilizing minimum curvature (k2) flow to account for lumen irregularities). In some embodiments, edge potential maps are calculated as outward-to-downward gradients in both contrast and non-contrast. In an exemplary embodiment, outer wall segmentation may utilize a cumulative distribution function (incorporating a prior distribution of wall thickness, e.g., distributions from 1-2 adjacent levels) on the velocity function to allow for median thickness in the absence of any other edge information. In an exemplary embodiment, the ferret diameter may be used for vessel characterization. In a further embodiment, wall thickness may be calculated as the sum of the distance to the lumen and the distance to the outer wall. In a further embodiment, lumen and/or wall segmentation may be performed using semantic segmentation, e.g., using a CNN.

Exemplary systems related to the assessment of vasculature can further advantageously analyze vascular composition. For example, in some embodiments, composition may be determined based on image intensity and other image features. In some embodiments, lumen shape can be utilized, for example, as related to determining thrombosis. Analyte blob models can be advantageously used to better analyze the composition of specific subregions of a vessel. Analyte blobs are defined as spatially contiguous regions of one class of biological analyte in a 2D, 3D, or 4D image. Blob models can utilize an anatomically aligned coordinate system, for example, in normalized radial distance from the luminal surface to the adventitial surface of the vessel wall, using isocontours. The model can advantageously identify one or more blobs and analyze each blob location, for example, with respect to the overall vascular structure and in comparison to other blobs. In exemplary embodiments, a hybrid Bayesian/Markov network can be utilized to model the relative location of the blobs. The model can advantageously explain the image intensity observed at pixels or voxels that are influenced by the local neighborhood of hidden analyte category nodes, thereby explaining the partial volumes and scanner point spread function (PSF). The model can further allow for dynamic bordering of analyte blob boundaries from the analyte probability map during inference by the analyzer module. This is an important difference from common machine vision approaches that pre-compute small regions to be analyzed, such as those using superpixel approaches, but cannot dynamically adjust these regions. An iterative inference procedure can be applied that utilizes current estimates of both analyte probability and blob boundaries. In some embodiments, parametric modeling assumptions or kernel density estimation methods can be used to allow probability density estimation among the sparse data used to train the model.

Presented here is a novel model for classification of the composition of vascular plaque components that removes the requirement for histology-to-radiology registration. The model still utilizes expert-annotated histology as a reference standard, but training of the model does not require registration to radiology imaging. The multiscale model computes statistics for each contiguous region of a given analyte type, which may be called a "blob." Within a cross section through a vessel, the wall is defined by two boundaries: an inner boundary with the lumen and an outer boundary of the vessel wall, creating a donut shape in the cross section. Within the donut-shaped wall region, there is a discrete number of blobs (unlike the default background class of regular wall tissue, which is not considered a blob). The number of blobs is modeled as a discrete random variable. Each blob is then assigned an analyte type label and various shape descriptors are computed. Additionally, the blobs are considered pairwise. Finally, within each blob, each pixel can generate a radiological imaging intensity value, which is modeled as an independent, identically distributed (i.i.d.) sample drawn from a sequentially evaluated distribution specific to each analyte type. Note that in this final step, the parameters of the imaging intensity distribution are not part of the training process.

One key feature of the model is that it considers the spatial relationships of analyte blobs within the vessel, recognizing that point-wise image features (whether from histology and/or radiology) are not the only source of information for an expert to determine plaque composition. The model can be trained without explicit histology-to-radiology registration, but can also be applied in situations where the registration is known. Statistically modeling the spatial layout of atherosclerotic plaque components for classification of unseen plaques is believed to be a novel concept.

Exemplary techniques for estimating vessel wall composition from CT or MR images are described in more detail in the next section. In particular, these methods may use multi-scale Bayesian analysis models. The basic Bayesian formulation is as follows:

In the context of this disclosure, the hypothesis can be based on a multi-scale vessel wall analyte map A with observation combing from CT or MR image intensity information I.

As depicted in Fig. 4, the multi-scale vessel wall analyte map may advantageously include wall-level segmentation 410 (e.g., cross-sectional slices of the vessel), blob-level segmentation, and pixel-level segmentation 430 (e.g., based on individual image pixels). For example, A = (B, C) may be defined as a map of vessel wall class labels (analogous to a graph with vertices B and edges C), where B is a set of blobs (cross-sectionally contiguous regions of non-background wall that share a label) and C is a set of blob pairs or pairs. _{B b} can be defined as a common single blob, where b ∈ [1...n _B ] is the index of all blobs in A. _{B b} ^a is a blob with label a. For statistical purposes, the individual blob descriptor operator D _B {} is in a low-dimensional space. C _c can be defined as a pair of blobs, where c ∈ [1...n B ] is the index of all blobs in A. n _B (n _B -1)/2] are the indices of all blob pairs in A. C _c ^f,g are blob pairs with labels f and g. For statistical purposes, the blob pair descriptor operator D _C {} is in a low-dimensional space. A(x)=a can be defined as the class label of pixel x, where a∈{'CALC', 'LRNC', 'FIBR', 'IPH', 'background'} (compositional property). In an exemplary embodiment, I(x) is the continuously valued pixel intensity at pixel x. Within each blob, I(x) is modeled as independent. Note that since the model is used to classify wall compositions in 3D radiological images, the word "pixel" is used to generically refer to both 2D pixels and 3D voxels.

Characterization of blob regions of similar composition/structure can advantageously provide insight into disease processes. Each slice (e.g., cross-sectional slice) of a vessel may advantageously include multiple blobs. Relationships between blobs can be evaluated pairwise. The number of blobs in a cross-section is modeled as a discrete random variable and may also have quantifiable significance. At the slice level of segmentation, relevant characteristics (e.g., biological properties/analytes) may include the total number of blobs of a particular structure/composition classification and/or a quantification of the number of blobs, relationships between blobs, e.g., spatial relationships such as closeness to the interior. At the blob level of segmentation, characteristics of each blob, e.g., structural properties such as size and shape, and compositional properties, may be evaluated to serve as biological properties/analytes. Finally, at the pixel level of segmentation, individual pixel-level analysis may be performed, e.g., the underlying image intensity distribution.

Probability mapping of properties may be applied with respect to the multi-scale vessel wall analyte map depicted in FIG. 4. Probability mapping may advantageously establish a vector of probabilities for all pixels with a component of the vector for the probability of each class of analyte and one component for the probability of background tissue. In an exemplary embodiment, the set of probability vectors may represent mutually exclusive properties. Thus, each set of probability vectors representing mutually exclusive properties sums to one. For example, in some embodiments, it may be known that a pixel should be classified into one, and only one, compositional category (e.g., a single coordinate of a vessel cannot be both fibrous and calcified). Of particular note is that probability mapping does not assume independence of analyte classes between pixels. This is because adjacent pixels or pixels within the same blob may typically have the same or similar properties. Thus, as described in more detail herein, the description of probability mapping advantageously accounts for dependencies between pixels.

f(A=α) can be defined as the probability density of map a. f(A) is the probability distribution function over all vessel walls. f(D _B {B ^a } = β) is the probability density of the descriptor vector β with label a. f(D _B {B ^a }) is the probability density function (pdf) of the blob descriptor with label a. For each value of a there is a probability distribution function. f(B) = Πf(D _B {B ^a }) f(D _C {C ^f,g } = γ) is the probability density of the pairwise descriptor vector γ with labels f and g. f(D _C {C ^f,g }) is the probability density function (pdf) of the pairwise blob descriptor. For each ordered pair f,g there is a probability distribution function. Thus:
f(C)=Π f(D _c {C ^a })
f(A)=f(B) f(C)=Π f(D _b {B ^a }) Π f(D _c {C ^a })
P(A(x)=a) is the probability that pixel x has label a. P(A(x)) is the probability mass function (pmf) (prevalence) of the analyte. It can be viewed as a vector of probabilities at a particular pixel x, or a probability map of a particular class label value.
Note that f(A) = P(N) * f(C) * f(B) = P(N) * Πf( _Cc ) * Πf( _Bb ).

f( _Cc = γ) is the probability density of the pairwise descriptor vector γ. f( _Cc ) is the probability density function (pdf) of the pairwise blob descriptors. f( _Bb = β) is the probability density of the descriptor vector β. f( _Bb ) is the probability density function (pdf) of the blob descriptors. P(A(x) = a) is the probability that pixel x has label a. P(A(x)) is the probability mass function (pmf) of the analyte (prevalence in a given map). It can be viewed as a vector of probabilities at a particular pixel x, or a spatial probability map of a particular analyte type. P(A(x) = a | I(x) = i) is the probability of the analyte given the image intensity, which is the main goal to calculate. P(I(x) = i | A(x) = a) is the distribution of image intensities for a given analyte.

Figure 5 depicts an example of a pixel-level probability mass function as a set of analyte probability vectors. As mentioned above, the following assumptions can inform the probability mass function: Completeness: In an exemplary embodiment, it can be assumed that a small enough pixel must be classified into at least one of the analyte classes (including a catch-all "background" category) such that the probabilities sum to one. Mutual exclusivity: A small enough pixel can be considered to belong to only one class of analyte. If there are combinations (i.e., spiky calcium in the LRNC background), new combination classes can be created to maintain mutual exclusivity. Non-independence: It can be assumed that each pixel is highly dependent on its neighbors and the overall structure of A.

An alternative way of looking at the analyte map is as a spatial map of the probability of a given analyte. At any given time during inference, a blob of the analyte can be defined using the full width at half maximum rule. With this rule, for each local maximum in the probability of that analyte, the region grows outward to a low threshold of half the local maximum. Note that this 50% value is an adjustable parameter. Here, we can do spatial regularization of the blobs (smooth with few topological holes) by implementing some curvature evolution in the probability map to keep the boundaries more realistic. Note that until we collapse the probabilities, different possible estimate blobs of different analyte classes may in general have spatial overlap, since they represent alternative hypotheses for the same pixel, and therefore the modifier "estimate".

Once the iterative inference is finished, there are several options for representing the results. First, the continuously valued probability map can be directly presented to the user in one of several forms, including (but not limited to) surface plots, contour plots, or using image fusion similar to visualizing PET values as changes in hue and saturation on CT. A second alternative is to collapse the probability map at each pixel by selecting one analyte label for each pixel. This can be done most simply by independently selecting the maximum posterior value at each pixel, thus creating a categorical map that can be visualized by assigning a different color to each analyte label and assigning full or partial opacity on top of the radiological image. In this second alternative, label values can be assigned non-independently by decomposing overlapping putative blobs based on the probability priorities of each blob. Thus, at a given pixel, the probability of a lower priority analyte may be used for the label if it belongs to a higher probability blob.

FIG. 6 illustrates a technique for computing putative analyte blobs. In an exemplary embodiment, the putative blobs may have overlapping regions. It may therefore be advantageous to apply analytical techniques to the segmentation of pixels by putative blobs. For a given analyte probability, a local maximum in the probability is determined. A full width at half maximum rule may then be applied to determine discrete blobs. In any given iteration of inference, the analyte blobs may be defined using the full width at half maximum rule. After finding the local maximum, grow the region with a low threshold of 0.5*max. (The value of 50% may be an adjustable parameter.) In some embodiments, spatial regularization of the blobs may also be applied, for example, by performing some curvature expansion on the probability map to keep the boundaries smooth and avoid holes. It should be noted that at this stage, different putative blobs of different analyte classes may generally overlap spatially, since they represent alternative hypotheses until the probability collapses. Thus, an image-level analyte map is computed, for example, based on the collapse of a probability map function. In particular, this collapsing can be determined based on either pixel-level analyte probability maps, estimated blobs, or a combination of both. For pixel-level analyte probability maps, collapsing can be determined for each pixel by selecting the label with the maximum probability A(x):=arg maxa P(A(x)=a). This is similar to the implementation Viterbi algorithm. Essentially, the highest probability is fixed for each set of mutually exclusive probability vectors (e.g., breaking possible ties with analyte priorities). All other probabilities in the set can then be set to zero. In some embodiments, the probabilities of neighboring pixels/regions may be considered when collapsing the data at the pixel level. For estimated blob-level collapsing, overlapping estimated blobs may be resolved. In some embodiments, the prioritization can be based on the blob probability density f(D ₁ {A _a ^b }=d ₁ ). This can impact the analysis of blob-level characteristics, since high probability blobs can change the shape of overlapping lower probability blobs. In an exemplary embodiment, the full range of probabilities can be maintained rather than collapsing the data.

To model the relative spatial arrangement of blobs within the vessel wall, an appropriate coordinate system can be chosen to provide rotational, translational, and scale invariance between different images. These invariances are important for the model because it can be trained on one type of vessel (e.g., carotid arteries, where endarterectomy specimens are readily available) and applied to other vascular beds (e.g., coronary arteries, where plaque specimens are not commonly available) under the assumption that the arteriosclerosis process is similar between different vascular beds. For tubular objects, the natural coordinate system continues from the centerline of the vessel, where the distance along the centerline provides the ordinate, and each plane perpendicular to the centerline has radial distance and angular polar coordinates. However, due to the variability of the vessel wall contour, especially in diseased patients that may be the subject of analysis, improved coordinate systems may be utilized. The longitudinal distance is calculated such that each 3D radiographic image pixel is given a value, not just along the centerline or an interpolated vertical plane. For a given plaque, the proximal and distal planes perpendicular to the centerline are each used to create an unsigned distance map on the original image grid, which are called P(x) and D(x), respectively, where x represents a 3D coordinate. The distance map l(x) = P(x)/(P(x) + D(x)) represents the relative distance along the plaque, with values 0 on the proximal plane and 1 on the distal plane. The direction of the l axis is determined by ∇l(x).

Because the wall shape can be significantly non-circular, the radial distance can be defined based on the shortest distance to the inner luminal surface and the shortest distance to the outer adventitial surface. Expert annotation of the tissue images includes regions that define the lumen and vessel (defined as the union of the lumen and vessel wall). Signed distance functions can be created for each of L(x) and V(x), respectively. The convention is to have the interior of these regions negative, and L positive and V negative at the wall. The relative radial distance is calculated as r(x) = L(x)/(L(x)-V(x)). It has a value of 0 at the luminal surface and a value of 1 at the adventitial surface. The direction of the r axis is determined by ∇r(x).

Because the wall contour is non-circular, the normalized tangential distance can be defined to lie along the contour of r (and 1 when working in 3D). The direction of the t axis is determined by ∇r × ∇l. By convention, the histological slice is assumed to be viewed from proximal to distal, with positive 1 pointing into the image. Note that unlike the others, t does not have a natural origin, since it wraps around the vessel. Therefore, the origin of this coordinate can be defined differently for each blob, relative to the blob's centroid.

Another wall coordinate used is the normalized wall thickness. In a sense, this is a proxy for disease progression. It is assumed that thicker walls are due to more advanced disease. The assumption is that the statistical relationships of the analytes change with more advanced disease. The absolute wall thickness can be calculated simply as w _abs (x) = L(x) - V(x). To normalize it to the range [0 to 1], one can determine the maximum possible wall thickness when the lumen approaches zero size, completely eccentric and close to the outer surface. In this case, the maximum diameter is the maximum ferret diameter of the vessel, _Dmax . Thus, the relative wall thickness can be calculated as w(x) = w _abs (x) / _Dmax .

The extent to which the aforementioned coordinates may be used in the model depends in part on the amount of training data available. If training data is limited, several options are available. Relative longitudinal distances can be ignored by treating the different segments through each plaque as if they were from the same statistical distribution. Plaque composition has been observed to vary along the longitudinal axis with more severe plaques appearing in the center. However, instead of parameterizing the distribution by l(x), this dimension can be collapsed. Similarly, relative wall thickness can also be collapsed. It has been observed that certain analytes occur in the "shoulder" regions of the plaque where w(x) will have intermediate values. However, this dimension can also be collapsed until sufficient training data is available.

As noted above, a vessel wall composition model can be utilized as an initial hypothesis (e.g., in the previous P(A)). Figure 7 illustrates normalized vessel wall coordinates for an exemplary vessel wall composition model. In the depicted model, 1 is the relative longitudinal distance along the vessel target from proximal to distal, which can be calculated, for example, in the normalized interval [0, 1]. The longitudinal distance can be calculated using two fast marching propagations starting from the proximal and distal faces to calculate the unsigned distance fields P and D, where 1 = P/(P + D). The l axis direction is ∇l. As depicted, r is the normalized radial distance, which can also be calculated in the normalized interval [0, 1] from the lumen to the adventitia surface. Thus, r = L/(L + (-V)), where L is the lumen signed distance field (SDF) and V is the vessel SDF. The r axis direction is ∇r. Finally, t is the normalized tangent distance, which can be calculated, for example, in the normalized interval [-0.5, 0.5]. In particular, in the exemplary embodiment, for individual analyte blobs only, there may not be a meaningful origin across the wall (so the origin may be at the blob centroid). The tangent distance is calculated along the 1 and r contour curves. The t axis direction is ∇r × ∇l.

Figure 9 shows some complex vascular topologies that can be accounted for using the techniques described herein. In particular, when processing CT or MR in 3D, different branches may be analyzed separately, such that relationships between analyte blobs in different branches are appropriately ignored. Thus, if a segmented image (cross-sectional slice) contains more than one lumen, this can be accounted for by dividing the wall into regions belonging to each lumen and then performing a watershed transform on r to consider/analyze each region separately.

As noted above, many of the coordinates and probability measures described herein can be expressed utilizing a normalized scale, thereby preserving scale invariance, for example, between vessels of different sizes. Thus, the proposed model may be advantageously independent of absolute vessel size, under the assumption that disease processes are similar and proportional for vessels of different calibers.

In some embodiments, the model may be configured to characterize concentric versus eccentric plaques. In particular, all normalized thicknesses close to 1 may indicate highly eccentric locations. In further embodiments, characterization of inner versus outer plaques may be implemented. In particular, histological information regarding this characteristic is hindered by deformation. Thus, in some embodiments, CT and training data may be utilized to establish an algorithm for determining inner versus outer plaque characteristics.

As noted above, in an exemplary embodiment, non-imaging data such as histological data can be utilized as a training set to establish algorithms that link image features to biological properties/analytes. However, there are some differences between the data types that need to be addressed to ensure proper correlation. For example, the following differences between histology and imaging can impact proper correlation: Carotid endarterectomy (CEA) leaves the adventitia and some media on the patient's CT or MR imaging analysis that is presumed to find the outer adventitial surface. (See, for example, FIG. 8 showing the margins between plaques removed for histological specimens against the outer vessel wall). In particular, the scientific literature indicates uncertainty as to whether calcification can occur in the adventitia. To account for this difference, the following techniques can be used: Histology can be extended outward, for example, based on the assumption that little or no analytes are left in the wall. Alternatively, image segmentation can be eroded inward, for example, based on knowledge of typical or specific margins left. For example, an average margin can be utilized. In some embodiments, the average margin may be normalized as a percentage of the total vessel diameter. In further embodiments, the histology may be used to mask the imaging (e.g., overlay based on alignment criteria). In such embodiments, it may be necessary to apply one or more transformations to the histology data to match the proper alignment. Finally, in some embodiments, the difference may be ignored (this is equivalent to uniformly scaling the removed plaque across the wall). This may induce some small errors, but perhaps the remaining wall may be thin compared to the plaque in CEA patients.

There may also be longitudinal differences between the histological data (e.g., training set) and the imaging data represented by the vessel wall composition model. In an exemplary embodiment, the longitudinal distance may be explicitly modeled/correlated. Thus, for example, the histological slice numbering (e.g., A-G) may be used to roughly determine the location within the excised portion of the plaque. However, this approach limits the analysis to other slices without corresponding histological data. Thus, alternatively, in some embodiments, all histological slices may be treated as arising from the same distribution. In an exemplary embodiment, some limited regularization may still be used along the longitudinal direction.

As noted above, normalized wall thickness is in some sense an imperfect proxy for disease progression. In particular, thicker walls are assumed to be due to more advanced disease, for example, based on the assumption that the statistical relationships of analytes change with more advanced disease. Normalized wall thickness can be calculated as follows: Determine absolute wall thickness T _a (in mm), which can be calculated, for example, as T _a =L+(-V), where L is the lumen SDF, V is the vessel SDF, and _Dmax is the maximum ferret diameter of the vessel (mm). Then, relative wall thickness T can be calculated as T=T _a /D _max , for example, based on the interval [0,1], where 1 indicates the thickest part of the small lumen, indicating a completely eccentric plaque. In an exemplary embodiment, the probabilities may be adjusted based on wall thickness, for example, such that the distribution of analyte blobs depends on the wall thickness. This can advantageously model differences in analyte composition over the course of disease progression.

10 depicts representing an exemplary analyte blob with a distribution of normalized vessel wall coordinates. In particular, the origin of t is placed at the blob centroid. The (r,t) coordinates are random vectors whose location/shape is fully represented by the joint distribution of points within. This can be simplified by considering the marginal distribution (since radial and tangential shape properties appear to be relatively independent). The marginal distributions can be calculated as projections along r and t (note that l and T coordinates can also be considered). In particular, the radial marginal distribution can advantageously represent/characterize plaque growth in concentric layers (e.g., medial, adventitial, and intimal layers). Similarly, the tangential marginal distribution can advantageously represent growth factors that may be indicative of disease staging. In an exemplary embodiment, the analyte blob descriptor can be calculated based on the marginal distribution. For example, one can obtain low-order statistics of the marginal distribution (or use histograms or fit parametric probability distribution functions).

In an exemplary embodiment, the following analyte blob descriptors may be used to capture, for example, the location, shape, or other structural characteristics of individual blobs:
- Position in normalized vessel coordinates - Mainly in terms of r - e.g. to distinguish between shallow and deep calcifications - t direction is ignored. [Optionally model the l direction]
- Range of normalized vessel coordinates - Range is a normalized value, we purposely avoid the word "size" which implies an absolute measurement - Bias expressing the degree of asymmetry of the distribution - Clinical significance unknown, but may be useful for regularizing shapes for implausible skewed shapes - Alignment to express confinement to parallel tissue layers - Analyte blobs seem to stay very well within the radial layers (r contours), so this may be useful for selecting similar imaged shapes - Wall thickness in which the blobs are located - Thick (i.e. advanced) plaques assumed to have different statistics than thin plaques

In some embodiments, pairwise blob descriptors may also be utilized, for example:
- Relative location - e.g. if fibrosis is on the luminal side of the LRNC - Relative extent - e.g. thickness/width of fibrosis compared to the LRNC - Circumference - how close one peripheral projection is to the middle of the other - e.g. napkin ring sign or fibrosis around the LRNC - Relative wall thickness - to express the degree of "shoulder" (shoulder is relatively thinner than the central plaque body)

Note that higher order interactions (e.g., between three blobs or between two blobs and another feature) can also be implemented, although diminishing returns and training limitations may be considered.

The following is an exemplary quantification of a blob descriptor:

In particular, the set of descriptors (e.g., 8-12 descriptors) forms a finite shape space in which blobs exist. The distribution of the population of blobs can then be viewed as a distribution in this finite space. Figure 11 illustrates an example distribution of blog descriptors. In an exemplary embodiment, the distribution of blob descriptors may be computed over the entire training set. In some embodiments, low-order statistics can be utilized for the individual blob descriptors (assuming independence) (e.g., where: E[α _r ], Var[α _r ]). In other embodiments, multi-dimensional Gaussian (mean vector + covariance matrix) analysis can be used to model the descriptors (e.g., when independence is not assumed). In a further embodiment, if the distribution is non-normal, density estimation techniques can be used to model it.

As mentioned above, one can model the number of blobs per cross section (or the number of each class), e.g., η without considering the class of analytes, and the number of η _i counts in each analyte class. Figure 14 shows the frequency distribution of the total number of blobs for each histological slide. Poison regression has been over-applied. Note that the analysis chart in Figure 14 depicts the number of blobs per cross section N without considering the analyte class (the number of blobs of each analyte type is represented by B).

To summarize the previous sections, in an exemplary embodiment, the global vessel wall composition model may include:
Analyte per pixel before pmf - P(A(x)=a _i )=ρ _i
Individual blob descriptors - B ₁ = (α _r , β _r , β _t , γ _r , γ _t , δ _r , δ _t , τ _T )
-B ₁ ~N (μ _1, Σ ₁ )
Pairwise blob descriptor - _C2 = ( _αrr , _αtt , _βrr , _βtt , _εrr , _εtt , _τTT )
-C ₂ ~N (μ ₂ , Σ ₂ )
・Number of blobs - η ~ Poission (λ _η )
Wherein:

As mentioned above, the imaging model may serve as the likelihood (e.g., P(I￥A)) of the Bayesian analysis model. A maximum likelihood estimate may then be determined. In an exemplary embodiment, this may be done by considering each pixel in isolation (e.g., without considering the prior probability of the structure in the model). The estimated analyte map is typically smooth only because the image is smooth (so, typically, no prior smoothing is performed). An independent pixel-by-pixel analysis may be performed, for example, at least up to the point of considering the PSF of the scanner. The imaging model is utilized to account for incomplete imaging data. For example, imaging small components of plaque adds independent noise on top of the pixel values. Furthermore, partial volume effects and the scanner PSF are well known to apply to small objects. Thus, given a model (e.g., a level set representation of the analyte region), simulating CT with Gaussian blur using the PSF is simple and fast. The imaging models described herein may also be applied to determine (or estimate) the distribution of true (unblurred) densities of different analytes. In particular, this density distribution cannot be obtained from a typical imaging study, as it would have blurred image intensities. In some embodiments, a wide variance can be used to represent the uncertainty. Alternatively, the distribution parameters can be optimized from a training set, but the objective function needs to be based on downstream readouts (of the analyte region), unless, for example, aligned histology data is available. FIG. 12 shows an exemplary model of imaging data (e.g., correlation between hidden (categorical) states (A(x)) and observed (continuous) states (I(x)), which accounts for random (e.g., analyte density distribution (H(A(x))) and deterministic (e.g., scanner blur*G(x)) noise factors. θ are the parameters of H (proportions of each analyte and mean/variance of HU). For N different analyte classes assuming normal distribution, θ=(τ ₁ , μ ₁ , σ ₁ , ..., τ _N , μ _N , σ _N ). Note that θ is patient-specific and is estimated with an expectation-maximization (EM) approach, e.g., where analyte labels are latent variables and images are observed data.
E-step: Determine the membership probability given the current parameters. M-step: Maximize the likelihood of the parameters given the membership probability.

FIG. 13 illustrates an exemplary Markov model/Viterbi algorithm diagram for association of observed and hidden states in an image model. In particular, the diagram illustrates observed states (gray) (observed image intensity, I(x)) and hidden states (white) (pure analyte intensity, H(A(x))), which are modeled with empirical histograms or with Gaussian or boxcar probability distribution functions. The PSF of the imaging system is modeled as a Gaussian, G(x). Thus, we have the following equation:
I(x)=G(x)*H(A(x))

An algorithm like Viterbi can be applied here, but note that the convolution will change the emission probabilities and H can be modeled as a Gaussian or uniform distribution.

As mentioned above, part of the inference procedure is based on expectation maximization (EM). In a typical application of EM, data points are modeled as belonging to one of several unknown classes. Each data point has a feature vector, and for each class, this feature vector can be modeled with a parametric distribution, such as a multidimensional Gaussian distribution, represented by a mean vector and a covariance matrix. In the context of the model presented here, a simple EM implementation works as follows:

The main problem with this simple model is that it does not code higher order structure into the pixels. There are no prior probabilities associated with more realistic arrangements of pixels. Only the tau determines the proportions of analyte classes. Therefore, the tau variable can be used to insert into a prior probability model of the blob, especially in the step of updating the membership probabilities.

Thus, a modified Bayesian inference procedure may be applied with much more sophisticated Bayesian priors. In the basic EM implementation, there is no real prior distribution. The variable tau represents the a priori relative proportions of each class, but even this variable is not specified and is estimated during the inference procedure. Thus, there is no a priori certainty about the distribution of classes in the basic EM model. In our model, the previous model is represented by a multi-scale analyte model. Tau becomes a function of location (and other variables) and not just the global proportions.

The membership probability function is defined as:

The inference algorithm is as follows: At each step of the iteration, the membership probability maps are initialized to zero, so that all class probabilities are zero. Then, for all possible model configurations, the membership probability maps can be added as follows:

Finally, the probability vector can be normalized at each pixel of the membership probability map to restore the assumption of completeness. Advantageously, it can be repeated over all model configurations. This is done by sequentially considering values of N from 0 to a relatively low value (e.g. 9), at which point it is observed that very few partitions have the same number of blobs. For each value of N, a different putative blob configuration can be examined. The putative blobs may be thresholded to a small number (N) based on their individual blob probabilities. Then, all permutations of the N blobs are considered. Thus, all the most likely blob configurations can be considered simultaneously, and each model can be weighted by a prior probability. This procedure is obviously an approximate inference scheme, since the entire space of multi-scale model configurations may not be considered. However, it can be assumed that a good approximation is achieved by considering the most likely ones (both in terms of N and blobs). It can also be assumed that in this procedure, a weighted average of the most likely configurations provides a good estimate at each individual pixel. Another alternative is to perform a constrained search of model configurations and select the most likely model as the MAP (maximum a posteriori) estimate.

Further exemplary statistical models (e.g., posterior P(A/I)) are also described herein. In CT angiography, the following information may be available:
Intensity - CT Hounsfield units or MR intensity - possibly other imaging features Location relative to anatomy - where in the plaque the pixel is Neighboring pixels - for example to smooth the contour via a level set

The posterior probability can be calculated as follows:

Thus, the following image information may affect the analyte probability Ai(x): I(x) is the observed image intensity (possibly a vector); T(x) is the relative wall thickness observed from image segmentation; F(x) is a CT image feature; S(x) is a feature of the vessel wall geometry (e.g., luminal bulging).

In some embodiments, an approach such as Metropolis-Hastings may be utilized. In other embodiments, a maximum a posteriori approach may be applied.

The following are exemplary algorithmic possibilities for the statistical analysis model. In some embodiments, the model may utilize belief propagation (aka max sum, max product, sum product messaging). Thus, for example, a Viterbi (HMM) type approach may be utilized, where the hidden states are the analyte assignments, A, and the observed states are the image intensities, I. This approach may advantageously discover that the MAP estimate may be argmax P(A|I). In some embodiments, a soft output Viterbi algorithm (SOVA) may be utilized. Note that the reliability of each decision may be indicated by the difference between the selected (survivor) path and the discarded path. This may therefore indicate the reliability of each pixel analyte classification. In further exemplary embodiments, a forward/backward Baum-Welch (HMM) approach may be utilized. For example, the most likely state at any point in time may be calculated, but the most likely sequence is not (see Viterbi).

Another possible technique is the Metropolis-Hastings (MCMC) approach, e.g., repeatedly sampling A and weighting it by the likelihood and prior. In some embodiments, a simple MRF version for sampling can be utilized. Note that directly sampling the posterior can be particularly advantageous. In an exemplary embodiment, a pixel-by-pixel histogram of analyte classes can be constructed.

Other algorithmic possibilities include application of Gibbs samplers, variational Bayes (similar to EM), mean field approximations, Kalman filters, or other techniques.

As mentioned above, in some embodiments, an expectation-maximization (EM) posterior approach can be utilized. In this approach, the observed data X are the imaging values, the unknown parameters θ are due to the analyte map (but not the analyte probabilities), and the latent variables Z are analyte probability vectors. One key feature of this approach is that the class memberships (Z) and the model parameters (θ) can be iteratively estimated between them, since they are each interdependent. However, because the analyte map separates the analyte probabilities, the approach can be modified such that the current class memberships do not need to affect the model parameters (as they are learned in the training step). Thus, essentially, EM learns the model parameters as it iterates over the current data. An exemplary implementation of the EM approach advantageously computes maximum likelihood iteratively, but assumes a flat prior.

Techniques for expressing longitudinal covariance are also provided herein. Due to the wide spacing of histological slices (e.g., 4 mm), sampling may not faithfully capture the longitudinal variation of the analyte. However, 3D image analysis is typically performed, and there is likely some true longitudinal covariance. The problem is that histological information is typically not provided for longitudinal covariance. Nevertheless, the exemplary statistical models disclosed herein can reflect slow longitudinal variations.

In some embodiments, a Markov model/chain can be applied. Figure 15 depicts an exemplary embedding example of a 1D Markov chain for text/DNA. Traditionally, when applying MRF Markov chains to images, it is common to have as low an order as possible. However, due to conditional independence (Markov property), higher order chains can be advantageous. Otherwise, the data may become too scrambled to be of any value. This is demonstrated by a 1D sampling of an exemplary Markov chain applied to text.
Uniform probability sampling output:
-earryjnv anr jakroyvnbqkrxtgashqtzifzstqaqwgktlfgidmxxaxmmhzmgbya mjgxnlyattvc rwpsszwfhimovkvgknlg ddou nmytnxpvdescbg k syfdhwqdrj jmcovoyodzkcofmlycehp
0th order Markov chain output:
-ooyusdii eltgotoroo tih ohnnattti gyagditghreay nm roefnnasos r naa euuecocrrfca ayas el s yba anoropnn laeo piileo hssio d idlif beeghec ebnniouhuehinely neiis cnitcwasohs ooglpyocp h trog 1
^First-order Markov chain output:
-icke inginatenc blof ade and jaloghe y at helmin by hem owery re jot j whede chrve blan ted sesourthegebe inaberens s ichath fie watt o
^Second-order Markov chain output:
-he ton th a s my caroodif flows an the er ity thayertione will ha m othe nre re cleara quichow mushing whe so mosing bloa ck abeenem used she sighembs inglis day p wer wharon the gladdid wor thatd k
・3rd ^order Markov chain output:
-es in angull o shoppinjust stees the kercourts allech is hot temal liked be weavy because in coy mrs hand room him rolio and ceran in that he mound a dish when what to bitcho way forget p

An exemplary first-order Markov chain of text probability tables is depicted in Figure 16. Note that such tables are exponentially sized in the following order:
D = degree of Markov chain N = number of characters Size = N ^D

Higher orders therefore lead to problems of dimensionality. Advantageously, histological samples have very high resolution. However, histological samples are not statistically independent, which can lead to overfitting, as we will explain in more detail later. In general, the more conditional dependencies that are modeled, the more predictive the model can be.

In an exemplary embodiment, instead of a 1D sequence such as a character, a 2D Markov Random Field (MRF) can be used for pixel values. Figure 17 depicts the conditional dependency of the first pixel (black) on neighboring pixels (gray). In an exemplary embodiment, the subpopulation may exploit symmetry to reduce the number of dependencies by half. In some embodiments, the pixel's value can be a simple image intensity, or a probability value for classification problems. Problems exist with the use of typical MRFs. Traditional MRFs are mostly restricted to nearest neighbor pixels that provide conditional dependencies that greatly reduce the specificity of the probability space represented, typically only black/white blobs for very general purpose segmentation/filtering, and very short range dependencies. However, whereas pixels are highly discretized, blobs can completely change the probability distribution by simply missing one pixel and dropping into the next. Thus, real image structures are much more continuous than typically considered using MRFs.

For this reason, the systems and methods of the present disclosure can advantageously utilize an inference procedure, e.g., a Bayesian rule of posterior α likelihood × prior (P(A/I)αP(I/A) × P(A)). Using a crossword-type analogy, the inference procedure implemented by the systems and methods of the subject application is like trying to OCR a crossword puzzle from a noisy scan. Knowledge (even incomplete knowledge of some squares may help inform unknown squares of the crossword puzzle. Efficiency is further improved by considering both vertical and horizontal directions simultaneously. In an exemplary embodiment, the inference procedure may be heuristic, e.g., initializing with an uninformed prior, solving easy ones first to provide clues for difficult ones that are solved later. Thus, the relatively easy detection of biological properties such as high density calcium can inform the presence of other analytes that are difficult to detect, such as lipids. Each step of the inference procedure can narrow the probability distribution of the unsolved pixels.

As noted above, a higher order Markov chain is desirable to obtain usable data. A drawback of utilizing a higher order Markov approach is that there may not be enough data to inform the inference process. In an exemplary embodiment, this problem can be addressed by utilizing density estimation methods such as Parzen windowing or by utilizing Kriging techniques.

To form an inference procedure, one may initialize with the unconditional prior probability of the analyte and then begin refining the probability using the highest level of evidence. For example, in some embodiments, an uncertainty width may be associated with each analyte probability estimate. In other embodiments, a approximation of 1/N may represent such uncertainty.

The term "Markov" is used loosely here, in particular because the proposed Markov implementation is not memoryless and attempts to explicitly model long-range (spatial) dependencies.

Since CT resolution is low compared to histology and plaque anatomy, in some embodiments it may be preferable to utilize a continuous space (time) Markov model rather than a discrete space (time). This may work well with a level set representation of the probability map as it works naturally well with sub-pixel interpolation. Since the analyte states are discrete, the model will be a discrete space model. However, if the analyte represents a continuous probability rather than the presence/absence of the analyte, the model will be a continuous space model.

Turning to lung-based applications, Table 4 below depicts exemplary biological attributes/analytes that may be utilized in relation to the hierarchical analysis framework for such applications.

In particular, the system may be configured to detect lung lesions. Thus, an exemplary system may be configured for whole lung segmentation. In some embodiments, the system may include the use of minimum curvature evolution to solve the problem of juxtapleural lesions. In some embodiments, the system may implement lung component analysis (vessels, lacerations, bronchi, lesions, etc.). Advantageously, a Hessian filter may be utilized to facilitate the lung component analysis. In some embodiments, the lung component analysis may further include pleural infiltration, for example as a function of the laceration geometry. In further embodiments, attachment to anatomical structures may also be considered. In addition to the lung component analysis, the described separate ground glass vs. solid analysis may also be applied. This analysis may include the determination of geometry features such as volume, diameter, sphericity, image features such as density and mass, and fractal analysis.

Fractal analysis may be used to infer squamous growth patterns. To perform fractal analysis on very small regions of interest, our method adaptively changes the support of the convolution kernel to restrict them to the region of interest (i.e., the lung nodule). Intersecting vessels/bronchi and non-lesion features may be masked out for the purposes of fractal analysis. This is done by applying an IIR Gaussian filter to the masked local neighborhood and normalizing using IIR blur binary masking. In some embodiments, the fractal analysis may further include determining porosity (based on the variance of the local mean). This may be applied with respect to the lung lesion, subparts of the lesion. In an exemplary embodiment, an IIR Gaussian filter or a circular neighborhood may be applied. In some embodiments, IIR may be utilized to calculate the variance. The average of local variances (AVL) may also be calculated, for example, as applied to the lung lesion. Similarly, the variance of the local variances may be calculated.

In an exemplary embodiment, both the structure and composition of the lesion can be calculated. Advantageously, calculating the lesion structure can utilize total volume measurements of thin sections, which can improve the calculation of size measurement changes. As part of the assessment of the lesion structure, measurements such as subsolid and ground glass opacity (GGO) volumes can also be determined. Turning to the composition of the lesion, tissue characteristics such as compaction, infiltration, proximity, and perfusion can be calculated, for example, which can reduce false positive rates compared to conventional analysis.

18, a further exemplary hierarchical analysis framework 1800 for the system of the present disclosure is depicted. FIG. 18 can be understood as an elaboration of FIG. 1 that uncovers more details regarding an exemplary intermediate processing layer of the hierarchical inference system. Advantageously, the hierarchical inference still flows from the imaging data 1810 to the underlying biological information 1820 to the clinical disease 1800. However, in particular, the framework 1800 includes multiple levels of data points for processing the imaging data to determine biological properties/analytes. At the pre-processing level 1812, physical parameters, registration transformations, and region segmentation may be determined. This pre-processed imaging information may then be utilized to extract image features at the next level of data points 1814, such as intensity features, shape, texture, and temporal characteristics. The extracted image features may then be utilized at level 1816 to fit one or more biological models to the imaged anatomical structures. Exemplary models include Bayesian/Markov net lesion substructure, fractal growth models, or other models as described herein. The biological model can advantageously serve as a bridge to correlate imaging features to underlying biological properties/analytes at level 1822. Examples of biological properties/analytes include anatomical structure, tissue composition, biological function, gene expression correlations, etc. Finally, at level 1832, the biological properties/analytes can be utilized to determine clinical findings associated with the pathology, including, for example, those related to disease subtypes, prognosis, decision support, etc.

Figure 19 is an exemplary application of phenotyping purposes in vascular therapy prescription, using the AHA-adopted Stary plaque typing system as an underlay, with in vivo determined types shown as color overlay. The left panel shows an example of labeling according to the likely dynamic behavior of plaque lesions based on the physical properties of the plaque lesions, and the right panel shows an example of using the classification results to guide patient treatment. An exemplary mapping is "I", "II", "III", "IV", "V", "VI", "VII", "VIII"], resulting in class_map = [occult, occult, occult, occult, occult, unstable, unstable, stable, stable]. This method is not tied to Stary. For example, the Virmani system ["calcified nodule", "CTO", "FA", "FCP", "healed plaque rupture", "PIT", "IPH", "rupture", "TCFA", "ULC"] has been used with class_map = [stable, stable, stable, stable, stable, stable, unstable, unstable, unstable, unstable], and other typing systems may perform similarly well. In an exemplary embodiment, the disclosed system and method may merge disparate typing systems, change the class map, or have other variations. In the case of FFR phenotypes, values such as normal or abnormal may be used, and/or numeric values may be used, for example to facilitate comparison with physical FFR.

Figure 20 is an example of another disease, such as lung cancer. In this example, a subtype of the mass can be determined to direct the most beneficial treatment for the patient based on the manifestation phenotype.

Since the CNN contains filters that extract spatial context that is not (only) contained in the area measurements of the analyte, the CNN is expected to outperform the reading vector classification. Despite the reduced training set, using the CNN may be practical for the following reasons.
1) The problems are generally simplified since there are relatively few classes corresponding to drastically different alternative therapies (not completely granular as is done in research assays requiring ex vivo tissue), e.g., three phenotypes for a classification problem, three risk levels for an outcome prediction/risk stratification problem.
2) Processing of analyte regions into false color regions, for example by level sets or other algorithm classes, performs a significant part of the image interpretation by producing segmentations and presenting classifiers on a simplified, but significantly augmented, dataset. Scalable pipeline stages reduce the dimensionality of the data (reducing the complexity of the problem that the CNN must decompose) while providing intermediate values that can be verified, thereby increasing the reliability of the entire pipeline.
3) Reformatting the data using a normalized coordinate system removes noise variations due to variables that do not substantially affect the classification (such as vessel size in the plaque phenotyping example).

To test this idea, a three-stage pipeline was constructed:
1) Semantic segmentation to identify which regions of the biomass fall into a particular class; 2) spatial unwrapping to convert the cross sections of the veins/arteries into rectangles; and 3) a trained CNN to read the annotated rectangles and identify their associated class (stable or unstable).

Without loss of generality, the exemplary systems and methods described herein can apply spatial unwrapping (e.g., training and testing CNNs with spatial unwrapping (unwrapped dataset) and without spatial unwrapping (donut dataset). Unwrapping has been observed to improve validation accuracy.

Semantic segmentation and spatial unwrapping:
First, the image volume is preprocessed. This preprocessing may include target initialization, normalization, and other preprocessing such as deblurring or restoration to form a region of interest that contains the physiological target to be phenotyped. The region of interest is a volume composed of cross sections through the volume. The body part is determined automatically or explicitly provided by the user. Body part targets that are tubular in nature are accompanied by a centerline. The centerline may branch, if present. The branches may be labeled automatically or by the user. A generalization regarding the concept of a centerline may be expressed for anatomical structures that are not tubular but benefit from some structural orientation, e.g., regions of tumors. In both cases, a centroid is determined for each cross section of the volume. For tubular structures, the centroid will be the center of a channel, e.g., the lumen of a blood vessel. For lesions, this will be the centroid of the tumor.

Figure 21 is a representation of an exemplary image pre-processing step, in this case deblurring or restoring using a patient-specific point spread determination algorithm to mitigate artifacts or image limitations resulting from the imaging process that may reduce the ability to determine features predictive of phenotype. This figure demonstrates a portion of a radiology application analysis of plaque from CT. Shown here is a deblurred or restored image that is the result of iteratively fitting a physical model of the scanner point spread function with normalized assumptions regarding the true underlying density of different regions of the image. This figure is included to illustrate that various image processing operations can be performed to aid in the ability to perform quantitative steps, and is not intended to indicate that this method is required for the particular invention of this disclosure, but rather is an example of steps that may be taken to improve overall performance.

The (optionally deblurred or restored) images are represented in an orthogonal dataset, where x represents the distance from the centroid, y represents the rotation theta, and z represents the cross-section. One such orthogonal set will be formed per branch or region. If multiple sets are used, "null" values will be used for overlapping regions, i.e., each physical voxel will be represented only once across the sets in such a way that they geometrically fit together. Each dataset will be paired with an additional dataset with sub-regions labeled by objectively verifiable tissue composition (see, e.g., FIG. 36). Examples of vascular tissue labels include lumen, calcification, LRNC, etc. Examples of lesion labels include necrotic, neovascularized, etc. These labels can be objectively verified, e.g., by histology (see, e.g., FIG. 37). The paired dataset will be used as input to a training step for building a convolutional neural network. Two levels of analysis are supported: analysis at the level of individual slices, optionally where the output varies continuously between adjacent slices, and analysis at the volumetric level (where individual slices may be considered as still frames, and the scan of the vascular tree is thought of as a movie).

Exemplary CNN Design:
AlexNet is a CNN that participated in the ImageNet Large Scale Visual Recognition Challenge in 2012. The network achieved a top-5 error of 15.3%. AlexNet was designed by the Supervision group, then consisting of Alex Krizhevsky, Geoffrey Hinton, and Ilya Sutskever at the University of Toronto. AlexNet was trained from scratch to classify an independent set of images (not used in the training and validation steps during network training). For the unwrapped data, an AlexNet-style network with 400x200 pixel input was used, and the donut network is an AlexNet-style with 280x280 pixel input (approximately the same resolution, but different aspect ratio). All convolution filter values were initialized with weights taken from AlexNet trained on the ImageNet dataset. Although the ImageNet dataset is a natural image dataset, this simply serves as an effective method of weight initialization. Once training begins, all weights are adjusted to better fit the new task. Most of the training schedule was taken directly from the open source AlexNet implementation, but some adjustments were required. Specifically, for both the AlexNet donut network and the AlexNet unwrapped network, the base learning rate was reduced to 0.001 (solver.prototxt) and the batch size was reduced to 32 (train_val.prototxt). All models were trained for 10,000 iterations and compared to snapshots when trained up to 2,000 iterations. Although a more detailed study on overfitting can be performed, in general, we found that both training and validation errors decreased between 2,000 and 10,000 iterations.

Alternative feature extensions (prefixes) may include the following:
・ResNet-https://arxiv. org/abs/1512.03385
・GoogleLeNet-https://www. cs. unc. edu/~wliu/papers/GoogLeNet. pdf
・ResNext-https://arxiv. org/abs/1611.05431
・ShuffleNet V2-https://arxiv. org/abs/1807.11164
・MobileNet V2-https://arxiv. org/abs/1801.04381

Runtime optimizations such as inter-frame redundancy between cross sections (sometimes called "temporal" redundancy, but in our case it is a form of inter-section redundancy) can be exploited to save computation (e.g., http://arxiv.org/abs/1803.06312). Many optimizations for training or inference can be implemented.

In an exemplary test implementation, AlexNet was trained to classify an independent set of images between two clinically significant categories, e.g., "unstable" and "stable" plaque, based on histological ground truth plaque types V and VI. The latter includes plaque types VII and VIII, which follow the industry de facto standard plaque classification nomenclature endorsed by the American Heart Association (AHA).

Without loss of generality, in the examples shown, both the overall accuracy and the confusion matrix were utilized to evaluate performance. This format was based on the concept of calculating four possibilities in a binary classification system: true positive, true negative, false positive, and false negative. Exemplary embodiments may utilize, for example, sensitivity and specificity as outcome variables, or F1 score (harmonic mean of accuracy and sensitivity), although other outcome variables may be used. Alternatively, AUC characteristics may be calculated for binary classifiers. Additionally, classifiers need not be binary based. For example, in some embodiments, classifiers may be sorted based on more than two possible states.

Expanding the dataset:
Because physician annotated data is expensive, it is desirable to artificially augment medical datasets (e.g., for use in training and/or validation). In the exemplary embodiment described herein, two different augmentation techniques were used: the donut was randomly flipped horizontally and rotated to a random angle between 0 and 360. The resulting rotated donut was cropped to the extent the donut was in, and then padded with black pixels to fill the image to have a square aspect ratio. The result was then scaled to a size of 280x280 and saved to a PNG.

The unwrapped dataset was expanded by randomly flipping it horizontally and "scrolling" it by a random number of pixels ranging from 0 to the width of the image. The result was then scaled to a size of 400x200 and saved to a PNG.

We augmented both datasets by a factor of 15, so that the total number of images after augmentation is 15 times the original number. We implemented class normalization, which means that the final datasets will have roughly the same number of images associated with each class. This is important because the original number of images for each class can differ, so the classifier will be biased towards classes with a higher number of images in the training set.

Without loss of generality, each radiologist performing the annotation may use any number of tissue types.

Figure 22 shows an exemplary application used to demonstrate aspects of the invention, in this case for classification of atherosclerotic plaque phenotype classification (using specific subject data as an example). Different colors represent different tissue analyte types, with dark grey representing normal wall. The figure shows the results of ground truth annotation of tissue features indicative of plaque phenotype, and the spatial context of how they are present in cross sections perpendicular to the axis of the vessel. The figure also shows the coordinate system developed to provide a common basis for the analysis of multiple histological cross sections. Grid lines are added to demonstrate the coordinate system (tangential distance vs. radial distance), overlaid on top of the color-coded pathologist annotations. The important aspect of this is that this kind of dataset may be efficiently used with deep learning approaches, since it simplifies the information using relatively simple false color images instead of high resolution full images, but without losing spatial context, and has formal representations for representations such as the "napkin ring sign", near luminal calcium, thin (or thick) cap (spacing between LRNC and lumen), etc.

Figure 23 shows tangential and radial variables represented internally using unit phasors, here with phasor angles coded in grayscale, illustrating the use of normalized axes for tubular structures relevant to blood vessels and other pathophysiology associated with such structures (e.g., the gastrointestinal tract). Note that the black to white vertical bars are purely arbitrary boundaries due to the grayscale encoding, and the normalized radial distance has a value of 0 at the luminal boundary and a value of 1 at the outer boundary.

Figure 24 shows an exemplary overlay of an exemplary radiology application that superimposes annotations generated from CTA (unfilled colored contours) over annotations generated by a pathologist from histology (solid areas). An exemplary aspect of the systems and methods presented herein is that the contours that form the in vivo non-invasive imaging can be used with a classification scheme to non-invasively determine phenotypes when the classifier is trained with a known ground truth. Specifically, filling in the contours shown unfilled in this figure (so as not to obscure the relationship with the ex vivo annotations for this particular segment that are provided to show the correspondence) creates input data for the classifier.

Figure 25 demonstrates a further step in data augmentation, specifically utilizing a normalized coordinate system to avoid extraneous variations associated with wall thickness and radial presentation. Specifically, the "donut" is "unwrapped" while preserving the pathologist's annotations. The left panel shows the pathologist's region annotations of a histological slice of plaque after morphing the cut-out "C" shape back to the in vivo "O" shape of the intact vessel wall. The horizontal axis is the tangential direction around the wall. The vertical axis is the normalized radial direction (lower is the luminal surface, upper is the outer surface). Also note that the finer granularity of the pathologist's annotations has been collapsed to match the granularity intended for extraction by in vivo radiology applications (e.g., LRNC, CALC, IPH). The right panel shows the comparable unwrapped radiology application annotations. Axes and colors are the same as the pathologist's annotations.

Figure 26 depicts the next refinement related to the plaque phenotyping example. Working from the unwrapped form, the lumen irregularities (e.g. as a result of ulceration or thrombus) and the locally varying wall thickness are depicted. The light grey at the bottom represents the lumen (added in this step to represent the irregularities of the luminal surface), and the black used in the previous step is now replaced by a dark grey to represent the varying wall thickness. The colours completely represent the outer regions of the wall.

Figure 27 depicts additional examples (using specific subject data as examples) to include aspects of, for example, intraplaque hemorrhage and/or other morphologies as desired. The left panel shows a representation of the donut, and the right panel shows it unwrapped with the luminal surface and local wall thickness.

Example CNNs tested included CNNs based on the AlexNet and Inception frameworks.

AlexNet results:
In the example embodiment tested, the convolution filter values were initialized with weights taken from AlexNet trained on the ImageNet (cited here) dataset. Although the ImageNet dataset is a natural image dataset, this simply serves as an effective way of initializing the weights. Once training begins, all weights are adjusted to better fit the new task.

Most of the training schedules were taken directly from the open source AlexNet implementation, but some adjustments were required. Specifically, for both the AlexNet donut network and the AlexNet unwrapped network, the base learning rate was reduced to 0.001 (solver.prototxt) and the batch size was reduced to 32 (train_val.prototxt).

All models were trained for 10,000 iterations and compared to snapshots when trained up to 2,000 iterations. A more detailed study on overfitting can be done, but in general we found that both training and validation errors decreased between 2,000 and 10,000 iterations.

An entirely new AlexNet network model was trained from scratch on four different combinations of ground truth results from two leading pathologists, two different ways of processing the images (see above), and on the unwrapped and donut images. The results are shown in Figure 28. For each dataset variation, the training data was augmented by 15x with class normalization enabled. The network was trained on this augmented data and tested on the corresponding unaugmented validation data corresponding to that variation. For the unwrapped data, an AlexNet-style network with 400x200 pixel input was used, and the donut network is AlexNet-style with 280x280 pixel input (approximately the same resolution, but different aspect ratio). Note that in the test embodiment, we changed the dimensions of the traditional and fully connected layers. Thus, the network of the AlexNet test embodiment can be described as a 5 convolutional, 3 fully connected layer network. Without loss of generality, some high-level conclusions that are implied from these results are presented below.
1) With the exception of the WN_RV dataset, the unwrapped data appears easier to analyze as it receives a higher validation accuracy overall; 2) The unnormalized data proves to be more representative, as expected.
3) Regarding the WN_RV dataset, the original idea was to pool the truth data of WN and RV to see the compatibility of the typing systems and the extent to which the sets could be merged. In doing so, significant differences were observed in the WN and RV data. The original purpose of the WN_RV experiment was to pool training data from multiple pathologists to see if that information would contribute to efficacy. Instead, a degradation rather than an improvement was observed. This was determined to be due to a change in the color scheme that prevented such data from being pooled. Thus, one could consider normalizing the color scheme to enable pooling.

Alternative network example: Start:
Transfer learning retraining of the Inception v3 CNN was started from the August 8, 2016 version of the network uploaded and published on the TensorFlow site. The network was trained for 10,000 steps. The training and validation sets were normalized in number of images by image augmentation so that both subsets had the same number of annotated images. All other network parameters were assumed to be default values.

A pre-trained CNN can be used to classify imaging features using the output from the last convolutional layer, which in the case of Google Inception v3 CNN is a numerical tensor of dimensions 2048x2. An SVM classifier is then trained to recognize objects. This process is typically performed on an Inception model after a transfer learning and fine-tuning step where the last softmax layer of a model initially trained on the ImageNet 2014 dataset is removed and retrained to recognize new categories of images.

Alternative embodiments:
FIG. 29 provides an alternative example for phenotyping potentially cancerous lung lesions. The leftmost panel shows the segmented lesion outline, separated by preprocessing into solid and semi-solid ("ground glass") subregions. The middle panel shows its location within the lung, and the rightmost panel shows it with a false color overlay. In this case, the three-dimensional nature of the lesion is likely to be considered important, so instead of processing two-dimensional slices separately, techniques such as video interpretation from computer vision may be applied to the classifier input dataset. Indeed, these methods can be generalized for tubular structures by processing multiple slices sequentially, such as a "movie" sequence along the centerline.

Another generalization is when the false colors are not chosen from a discrete palette, but instead have continuous values at the pixel or voxel location. Using the lung example, Figure 30 shows a set of features, sometimes called "radiomics" features, that can be calculated for each voxel. Such a set of values may be present in any number of pre-processed overlays and may be fed into a phenotypic classifier.

Other alternative embodiments include using change data, e.g., collected from multiple time points, rather than (only) data from a single time point. For example, if the amount or quality of a negative cell type increases, it can be said to be a "progressor" phenotype, as opposed to a "regressor" phenotype if it decreases. A regressor may be due, for example, to a response to a drug. Alternatively, if the rate of change, e.g., of the LRNC, is fast, this may mean a different phenotype. Extensions of the examples to use delta values or rates of change will be apparent to those skilled in the art.

In additional alternative embodiments, non-spatial information, such as derived from other assays (e.g., laboratory results), demographics/risk factors, or other measurements taken from radiology images, may be fed into the final layer of the CNN to combine the spatial information with the non-spatial information. Also, localized information, such as the use of pressure lines with readings at one or more specific locations along the vessel from a reference such as a bifurcation or anastomosis, may be determined by inference of the full 3D coordinates at the time of imaging.

Although the focus of these examples has been on phenotyping, as a further embodiment of the present invention, a similar approach can be applied to the problem of outcome prediction.

Exemplary implementation:
The systems and methods of the present disclosure may advantageously include a multi-stage pipeline. Figure 34 provides a further exemplary implementation of a hierarchical analysis framework. Biological features are identified and quantified by semantic segmentation to identify, in an exemplary application, lipid-rich necrotic core, cap thickness, stenosis, dilation, remodeling ratio, tortuosity (e.g., entrance and exit angles), calcification, IPH, and/or ulceration, alone or in combination. The biological properties are represented numerically and in an augmented dataset, which involves spatial unwrapping by transforming vein/artery cross sections into rectangles and then transforming disease states (e.g., g, ischemia-inducing fractional flow reserve ratio FFR, high-risk phenotype HRP, and/or risk stratification time to event (TTE)) and reading the augmented dataset using trained CNN(s) to identify and characterize the condition. Patient images are collected and the raw slice data is used with a set of algorithms to measure biological properties that can be objectively verified, which are then formed into an augmented dataset to feed one or more CNNs, and in this example the results are used to implement constraints or continuous conditions. A recursive CNN is used to propagate forward and backward to create constraints (e.g., monotonically decreasing fractional flow reserve from proximal to distal throughout the vascular tree, or constant HRP values at focal lesions, or other constraints). Ground truth data for HRP may exist as an expert pathologist's determination of plaque type at a given cross-section determined ex vivo. Ground truth data for FFR is from a physical pressure line, with the pressure line having one or more measurements, with the network training such that the location along the centerline proximal to a given measurement is constrained to be greater than or equal to the measurement, the distal location is constrained to be less than or equal to the measurement, and if two measurements on the same centerline are known, the value between the two measurements is constrained to be within an interval.

These characteristics and/or conditions may be assessed at a given time point and/or may change over time (longitudinal). Without loss of generality, other embodiments that perform similar steps in either plaque phenotyping or other applications are embodiments of the present invention.

In an exemplary implementation, the biological characteristics may include one or more of the following:
● Angiogenesis ● Neovascularization ● Inflammation ● Calcification ● Lipid deposits ● Necrosis ● Hemorrhage ● Ulceration ● Stiffness ● Density ● Stenosis ● Dilation ● Remodeling ratio ● Tortuosity ● Flow (e.g., blood flow in a channel)
Pressure (e.g. blood in a channel or one tissue pressing against another)
Cell type (e.g. macrophages, etc.)
Cell alignment (e.g., of smooth muscle cells)
Shear stress (e.g. blood in a channel)

The analysis may include determining one or more of the amount, degree, and/or characteristics for each of the aforementioned biological properties.

Conditions that can be determined based on biological properties can include one or more of the following:
Perfusion/ischemia (when restricted) (e.g., brain or cardiac tissue)
Perfusion/infarction (during amputation) (e.g., brain or heart tissue)
● Oxygenation ● Metabolism ● Fractional flow reserve (perfusion capacity), e.g., FFR(+) vs. FFR(-) and/or contiguous counts ● Malignancy ● Invasion ● High-risk plaques, e.g., HRP(+) vs. HRP(-) and/or labeled phenotypes ● Risk stratification (as probability of event or as time to event) (e.g., MACCE, explicitly mentioned)

Fact-based forms of verification may include:
● Biopsy ● Expert tissue annotation forms of excised tissue (e.g. endarterectomy or autopsy)
Expert phenotypic annotation of excised tissue (e.g., endarterectomy or autopsy)
● Physical pressure lines ● Other imaging modalities ● Physiological monitoring (e.g. ECG, SaO2, etc.)
Genomic and/or proteomic and/or metabolomic and/or transcriptomic assays Clinical outcomes

Analysis can be performed at a given point in time or longitudinally (i.e., changes over time).

Exemplary system architecture:
31 shows a high level diagram of users and other systems interacting with the analytics platform in accordance with the systems and methods of the present disclosure. Stakeholders in this diagram include system administrators with interoperability, security, failover and disaster recovery, and regulatory concerns, and support technicians.

The platform can be deployed in two main configurations: on-premise or on a remote server (Figure 32). The platform deployment may be a standalone configuration (top left), an on-premise server configuration (bottom left), or a remote server configuration (right). The on-premise deployment configuration can include two sub-configurations: desktop only or rack-mounted. In the remote configuration, the platform can be deployed in a HIPAA compliant data center. Clients access its API server through a secure HTTP connection. The clients can be desktop or tablet browsers. No hardware is deployed at the consumer site other than the computer running the web browser. The deployed server may be on a public cloud or on an extension of the consumer's private network using a VPN.

An exemplary embodiment consists of a client and a server. For example, Figure 33 shows the client as a C++ application and the server as a Python application. These components interact using HTML 5.0, CSS 5.0, and Java Script. Open standards are used wherever possible for interfaces, including but not limited to HTTP(S), REST, DICOM, SPARQL, and JSON. Third party libraries are also used, as shown in this diagram showing the major parts of the technology stack. Many variations and different approaches may be appreciated by those skilled in the art.

Various embodiments of the above systems and methods may be implemented in digital electronic circuitry, computer hardware, firmware, and/or software. Implementations may be in a computer program product (i.e., a computer program tangibly embodied in an information carrier). Implementations may be in a machine-readable storage device and/or a propagated signal, for example, for execution by or to control the operation of a data processing apparatus. Implementations may be in, for example, a programmable processor, a computer, and/or multiple computers.

The computer program may be written in any type of programming language, including compiled and/or interpreted languages, and the computer program may be deployed in any form, including as a stand-alone program or as a subroutine, element, and/or other unit suitable for use in a computing environment. The computer program may be deployed to run on one computer or on multiple computers at one site.

The method steps may be performed by one or more programmable processors executing a computer program to perform the functions of the invention by manipulating input data to generate output. The method steps may also be performed by an apparatus implemented as special purpose logic circuitry, which may be implemented as special purpose logic circuitry. The circuitry may be, for example, an FPGA (field programmable gate array) and/or an ASIC (application specific integrated circuit). Modules, subroutines, and software agents may refer to portions of computer programs, processors, special circuitry, software, and/or hardware that implement the functionality.

Processors suitable for executing a computer program include, by way of example, both general purpose and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer can be operatively coupled to receive data from and/or transfer data to one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data.

The transmission of data and instructions can also take place via a communications network. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including, by way of example, semiconductor memory devices. The information carriers can be, for example, EPROM, EEPROM, flash memory devices, magnetic disks, internal hard disks, removable disks, magneto-optical disks, CD-ROM and/or DVD-ROM disks. The processor and memory can be supplemented by and/or incorporated in special purpose logic circuitry.

To provide for user interaction, the techniques described above may be implemented on a computer having a display device. The display device may be, for example, a cathode ray tube (CRT) and/or a liquid crystal display (LCD) monitor. User interaction may be, for example, a display of information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer (e.g., interact with user interface elements). Other types of devices may be used to provide user interaction. The other device may be, for example, feedback provided to the user in any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback). Input from the user may be received in any form, including, for example, acoustic, speech, and/or tactile input.

The above techniques can be implemented in a distributed computing system that includes a back-end component. The back-end component can be, for example, a data server, a middleware component, and/or an application server. The above techniques can be implemented in a distributed computing system that includes a front-end component. The front-end component can be, for example, a client computer having a graphical user interface, a web browser through which a user can interact with the exemplary implementation, and/or other graphical user interface for a transmission device. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network (LAN), a wide area network (WAN), the Internet, a wired network, and/or a wireless network.

The system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship.

Examples of packet-based networks include the Internet, carrier Internet Protocol (IP) networks (e.g., local area networks (LANs), wide area networks (WANs), campus area networks (CANs), metropolitan area networks (MANs), home area networks (HANs)), private IP networks, IP private branch exchanges (IPBXs), wireless networks (e.g., radio access networks (RANs), 802.11 networks, 802.16 networks, general packet radio service (GPRS) networks, HiperLANs), and/or other packet-based networks. Examples of circuit-based networks include the public switched telephone network (PSTN), private branch exchanges (PBXs), wireless networks (RANs, Bluetooth, code division multiple access (CDMA) networks, time division multiple access (TDMA) networks, Global System for Mobile Communications (GSM) networks), and/or other circuit-based networks.

Computing devices include, for example, computers, computers with browser devices, telephones, IP telephones, mobile devices (e.g., cell phones, personal digital assistant (PDA) devices, laptop computers, email devices), and/or other communication devices. Browser devices include, for example, computers (e.g., desktop computers, laptop computers) with World Wide Web browsers (e.g., Microsoft® Internet Explorer® available from Microsoft Corporation, Mozilla® Firefox available from Mozilla Corporation). Mobile computing devices include, for example, Blackberry®, iPAD®, iPhone®, or other smart phone devices.

Although many variations and modifications of the present disclosure will no doubt become apparent to those skilled in the art after reading the foregoing description, it should be understood that the specific embodiments shown and described by way of illustration are not intended to be considered limiting. Moreover, while the subject matter has been described with reference to specific embodiments, variations within the spirit and scope of the present disclosure will occur to those skilled in the art. It should be noted that the foregoing examples are provided merely for illustrative purposes and are not to be construed as limiting the present disclosure.

Although the present disclosure has been described herein with reference to specific embodiments, the disclosure is not intended to be limited to the details disclosed herein, but rather, the disclosure extends to all modifications and generalizations thereof that would be apparent to one skilled in the art, including those within the scope of the appended claims.

Claims (10)

1. A method for computer-aided phenotyping or outcome prediction of a lesion using an augmented radiological dataset, the method comprising:
receiving a non-invasively acquired radiology data set of a patient;
augmenting the data set by performing analyte measurements and/or classifications of anatomical structures, anatomical shapes or contours, tissue properties, tissue types, tissue features, or any combination thereof, with objective validation against a set of analytes associated with the lesion;
and processing the augmented dataset using a machine learning classification approach based on known ground truth to determine one or both of: (i) a phenotype of the lesion; or (ii) a predicted outcome associated with the lesion;
augmenting the dataset further comprises spatial transformation of the dataset to highlight biologically significant spatial context;
and said transforming includes spatially transforming relative to a cross-section of a lesion-specific structure within said radiation dataset to generate a transformed dataset suitable for the lesion;
the machine learning classification approach is the use of a trained convolutional neural network (CNN) applied to a transformed dataset appropriate for the lesion;
an image volume within the radiation data set is preprocessed to form a region of interest that includes a physiological target, a lesion, a set of lesions, or any combination thereof to be analyzed, the region of interest including one or more slices, each of the slices being constructed from a projection through the volume;
A method in which, if the region of interest comprises a volume that is directional in nature, the centerline is depicted for that volume such that the cross sections are taken along a centerline and a centroid is determined for each cross section, and the pre-processed radiation data set is represented in a Cartesian coordinate system in which axes are used to represent perpendicular distance from the centerline and rotation theta around the centerline, respectively. The method of claim 1 , wherein the region of interest includes multiple branches in a branched lesion-specific network with corresponding directionality, and a different Cartesian coordinate system is applied to each branch. The method of claim 2 , wherein any initially overlapping sections of the branches are assigned to a single branch. The method of claim 1, wherein, if the region of interest comprises a volume that is directional in nature, the centerline is plotted for that volume such that the slices are taken along a centerline and a centroid is determined for each slice, and subregions within each slice are classified based on objectively verifiable characteristics based on tissue composition. The method of claim 4 , wherein the tissue composition classification comprises classification of lesion-specific tissue properties, alone or in any combination. The method of claim 4 , wherein the tissue composition classification further comprises an intralesional hemorrhage (IPH) classification. The method of claim 4 , wherein the subregions are further classified based on abnormal morphology. The method of claim 7 , wherein classification of abnormal morphology comprises lesion identification, classification, or any combination thereof. 1. A method for computer-aided phenotyping or outcome prediction of a lesion using an augmented radiological dataset, the method comprising:
receiving a non-invasively acquired radiology data set of a patient;
augmenting the data set by performing analyte measurements and/or classifications of anatomical structures, anatomical shapes or contours, tissue properties, tissue types, tissue features, or any combination thereof, with objective validation against a set of analytes associated with the lesion;
and processing the augmented dataset using a machine learning classification approach based on known ground truth to determine one or both of: (i) a phenotype of the lesion; or (ii) a predicted outcome associated with the lesion;
augmenting the dataset further comprises spatial transformation of the dataset to highlight biologically significant spatial context;
and said transforming includes spatially transforming relative to a cross-section of a lesion-specific structure within said radiation dataset to generate a transformed dataset suitable for the lesion;
the machine learning classification approach is the use of a trained convolutional neural network (CNN) applied to a transformed dataset appropriate for the lesion;
Dataset augmentation includes providing ground truth annotations of analyte sub-regions and a spatial context for how such analyte sub-regions are present in the cross-sections, said spatial context including providing a coordinate system based on polar coordinates for a centroid of each cross-section;
The method, wherein the coordinate system is normalized by normalizing the radial coordinates to a tubular structure. 1. A method for computer-aided phenotyping or outcome prediction of a lesion using an augmented radiological dataset, the method comprising:
receiving a non-invasively acquired radiology data set of a patient;
augmenting the data set by performing analyte measurements and/or classifications of anatomical structures, anatomical shapes or contours, tissue properties, tissue types, tissue features, or any combination thereof, with objective validation against a set of analytes associated with the lesion;
and processing the augmented dataset using a machine learning classification approach based on known ground truth to determine one or both of: (i) a phenotype of the lesion; or (ii) a predicted outcome associated with the lesion;
augmenting the dataset further comprises spatial transformation of the dataset to highlight biologically significant spatial context;
and said transforming includes spatially transforming relative to a cross-section of a lesion-specific structure within said radiation dataset to generate a transformed dataset suitable for the lesion;
the machine learning classification approach is the use of a trained convolutional neural network (CNN) applied to a transformed dataset appropriate for the lesion;
A method in which the dataset is augmented by the visual use of different colors to represent different analyte sub-regions, and the color-coded analyte regions are visually depicted against an annotated background that distinguishes between: (i) a central luminal region inside the inner lumen of the lesion-specific target, lesion, set of foci, or any combination thereof being visualized; (ii) a non-analyte sub-region of the lesion-specific target, lesion, set of foci, or any combination thereof; and (iii) an outer region outside the outer wall of the lesion-specific target, lesion, set of foci, or any combination thereof.

Images