JP7780203B2 - SYSTEM AND METHOD FOR PROVIDING A PRIVATE MULTIMODAL ARTIFICIAL INTELLIGENCE PLATFORM - Patent application - Google Patents

Description

PRIORITY CLAIM This application claims priority to U.S. Provisional Application No. 63/020,930 (Docket No. 213-0104P), filed May 6, 2020, the contents of which are incorporated herein by reference. This application is a continuation-in-part of U.S. Application No. 16/828,085 (Docket No. 213-0100), filed March 24, 2020, which claims priority to U.S. Provisional Application No. 62/948,105, filed December 13, 2019, the contents of which are incorporated herein by reference. This application is a continuation-in-part of U.S. Application No. 16/828,216 (Docket No. 213-0101), filed March 24, 2020, which claims priority to U.S. Provisional Application No. 62/948,105, filed December 13, 2019, the contents of which are incorporated herein by reference. This application is a continuation-in-part of U.S. Application No. 17/176,530 (213-0102-CON) filed February 16, 2021, which is a continuation-in-part of U.S. Application No. 16/828,354 (213-0102) filed March 24, 2020, now U.S. Patent No. 10,924,460, issued February 16, 2021, and claims priority to U.S. Provisional Application No. 62/948,105, filed December 13, 2019, the contents of which are incorporated herein by reference. This application is a continuation-in-part of U.S. Application No. 16/828,420 (Docket No. 213-0103), filed March 24, 2020, and claims priority to U.S. Provisional Application No. 62/948,105, filed December 13, 2019, the contents of which are incorporated herein by reference.

This disclosure generally relates to training neural networks and introduces new techniques for training and deploying neural networks or other trained models in a manner that protects training data from various sources from being discoverable.

There are existing approaches for training neural networks that use federated or centralized training approaches. Each existing approach for training neural networks relies on data received from various clients. The process of sharing data in this context can lead to the data being leaked or becoming discoverable.

For example, deep learning and machine learning in healthcare contexts may require large datasets to produce sufficient accuracy in trained models to be diagnostically meaningful. Such data can include data on X-rays or MRIs across multiple patients or patient categories. One approach to training such models is to pool the raw data, and analytics can access a central repository of the pooled data to perform machine learning on large datasets. However, this approach presents ethical and privacy concerns, as well as other issues such as single points of failure and protection requirements.

To explain how the above-mentioned and other advantages and features of the present disclosure can be obtained, a more particular description of the principles briefly described above will be rendered by reference to specific embodiments that are illustrated in the accompanying drawings. With the understanding that these drawings depict only exemplary embodiments of the present disclosure and therefore should not be considered limiting of its scope, the principles herein will be described and explained with additional specificity and detail through the use of the accompanying drawings.

Figure 1 shows the training approach for the associative learning model. FIG. 2 illustrates the split-learning focused model training approach. FIG. 3 illustrates a partitioned learning peer-to-peer approach. FIG. 4 illustrates the federated split learning approach. FIG. 5 shows an embodiment for blind learning. FIG. 7 shows how blind correlation works across multiple clients. FIG. 8 illustrates an embodiment of the method. FIG. 9 illustrates an embodiment of the method. FIG. 10 illustrates an embodiment of the method. FIG. 11 shows an embodiment of the system.

Specific aspects and embodiments of the present disclosure are provided below. As will be apparent to one of ordinary skill in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and description are not intended to be limiting.

The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an effective description for implementing the exemplary embodiments. It should be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the present application, as set forth in the appended claims.

What is needed in the art is a method and system that combines known approaches for training neural network models that keeps the data on which the models are trained private. Previous approaches have the potential for training data to be leaked or discovered as part of the training process. The improved approach disclosed herein addresses other issues, such as protection concerns, and can eliminate single points of failure. This disclosure first describes known approaches and then introduces new approaches. In one aspect, a particular platform is used to enable federated development or training of neural network models. Use of the disclosed platform for training models in this manner is disclosed as another embodiment herein. In yet another embodiment, data is encrypted as it passes between a server and one or more client devices. Various types of federated learning (shown in FIG. 1), partitioned learning (shown in FIG. 2), and partitioned learning peer-to-peer (shown in FIG. 3) are disclosed herein. This disclosure provides several novel improvements over conventional approaches.

In typical federated learning, an entire model is passed from a server to a client device for training using client data. This process may involve using multiple different clients, each with its own data, for training purposes. This approach is typically performed in a linear, iterative manner, where the entire model is sent along with its data to a first client, and after training on the first client, the entire model is returned to the server for "averaging." The entire updated model is then sent to a second client along with its data for further processing. The updated model is then sent back to the server for further "averaging," and so on. In a partitioned learning approach, the model is partitioned and a portion is sent to each client, but there is still an inefficient, linear, iterative training process. Partitioned learning peer-to-peer approaches also perform linearly, as peer clients share data in a linear process. Improvements are needed to maintain data privacy and streamline the training process.

This disclosure describes two main improvements to federated and split learning. The first is a federated split learning approach (shown in Figures 4-5) in which client-side processing occurs in parallel, independent of other clients. The second disclosed approach (shown in Figures 6-10) relates to a multi-modal artificial intelligence (MMAI) training approach for processing different types of data from different clients.

As mentioned above, a federated split learning approach is disclosed as a variation of the exemplary federated learning approach described above. The method in this regard includes splitting a neural network into a first portion and a second portion at a server and separately transmitting the second portion to a first client and a second client. The clients may have data (MRIs, patient data, customer banking data, etc.), and each receives a portion of the neural network (a certain number of layers of the network up to the cut layer). The method includes performing the following operations until a threshold is met: (1) simultaneously performing a forward step on the second portion at the first client and the second client to generate data SA1 and SA2 (see FIGS. 1-4 ); (2) sending SA1 and SA2 from the first client and the second client to the server; (3) calculating loss values for the first client and the second client at the server; (4) calculating an average loss across the first client and the second client at the server; (5) performing backpropagation using the average loss to calculate gradients at the server; and (6) sending the gradients from the server to the first client and the second client. This approach offers an improvement over federated and partitioned learning approaches by allowing client-side (or “data server”) processing to operate independently and in parallel. This approach also differs from the peer-to-peer approach of partitioned learning. The independent data server sends the activations to the server side, which aggregates, averages, or otherwise processes the data depending on the network requirements to obtain the final trained model.

Another aspect of the present disclosure relates to improvements in developing artificial intelligence models that can utilize multiple different modes or types of data for training. For example, different clients may have different types of data. One client may have X-ray or MRI images, while another client may have text describing a patient's health condition. In this regard, a method may include dividing a neural network into a first client-side network, a second client-side network, and a server-side network, and transmitting the first client-side network to the first client. The first client-side network is configured to process first data from the first client, the first data having a first type. The first client-side network may include at least one first client-side layer. The method may include transmitting a second client-side network to a second client. The second client-side network is configured to process second data from the second client, the second data having a second type. The second client-side network may include at least one second client-side layer, the first type and the second type having a common association.

The method may further include receiving, at the server-side network, first activations from training a first client-side network on first data from a first client; receiving, at the server-side network, second activations from training a second client-side network on second data from a second client; training at least one server-side layer of the server-side network based on the first activations and the second activations to generate gradients; and transmitting the gradients from the server-side network to the first client-side network and the second client-side network. In this manner, a model is trained using multiple different types of data that have a common relationship, such as relating to a single patient or a single type or category of patients.

This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. This subject matter should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

Disclosed herein is a new system, platform, computing environment, cloud environment, marketplace, or other characterization of a system that enables an improved approach to training neural networks. In one aspect, this approach, referred to as a federated partitioned learning approach, combines features of known approaches but provides a training process that maintains the privacy of data used to train the model from various client devices. This disclosure first describes the federated learning approach in more detail, followed by a partitioned learning approach and a partitioned learning peer-to-peer approach, and then introduces a novel federated partitioned learning approach. Additionally, a multimodal artificial intelligence (MMAI) learning approach for various types of data is also introduced. The novel federated partitioned learning and MMAI approaches are based on several models, including those described above. This application reviews these first approaches in more detail and then introduces two novel learning techniques.

Federated Learning. Figure 1 illustrates a federated learning approach 100, which is currently being adopted by major companies. A drawback of this approach is that it proceeds "linearly," one data provider at a time, rather than in parallel. The example neural network shown is a fully connected, forward-propagation neural network being trained using a federated learning approach. The training process in this case involves a server 102 creating a model 104 and sharing models 106A, 108A, and 110A in a linear fashion with each of the clients 106, 108, and 110. The clients train each of the models 106A, 108A, and 110A separately as they receive them in turn, and each sends the trained model data back to the server 102, as shown. The server 102 averages the models and generates a new model 104 with updated weights (also known as the trained model). The server 102 then sends the new model or weights to each of the clients 106, 108, and 110 in a linear fashion. This process is repeated iteratively a number of times and until a particular accuracy is achieved.

In each iteration, the server 102 averages all participating models to create a trained model B. Thus, at any point in time, the server has a fully trained model 104. The term "global model" refers to the model resulting from the training process. The global model is the trained object that is used for the inference task. The inference task is to evaluate medical images to classify whether a patient has cancer, a fracture, or some other pathology.

An example of how this approach is used is when devices such as electronic watches or mobile devices, e.g., devices that are charged overnight and connected to a Wi-Fi network, use their processors to train neural network models. Thus, Client 1 (106) could be an Apple Watch, while Client 2 (108) could be someone else's iPhone. An example of a model is the Siri voice processing service provided by Apple. All devices train the same model; the only difference is that each client trains it with local data. The model or data is sent back to the server 102, which averages the models together. The drawback is that each client, such as Client 1 (106), could be tricked into sharing something about the data used to train the model. This is a privacy data leak, resulting in the problems mentioned above. The challenge with the federated learning approach is the lack of model privacy, since the entire model is passed from client to client. It is computationally expensive for each client to process the entire model, and communication overhead is significant because the entire model is sent multiple times. Reconstruction attacks can also make the training data vulnerable.

Partition Learning Figure 2 shows a partition learning-focused approach. A model (neural network) 204 is partitioned into two parts: one part (206A, 208A, 210A) resides on the client side 206, 208, 210, respectively, and includes only the input layer to the model and optionally other layers up to the cut layer; the other part (B) resides on the server side 202, and often includes the output layer. The partition layer (S) refers to the layer where A and B are partitioned (the cut layer). In Figure 2, SA represents the partition layer or data sent from A to B, and SB represents the partition layer sent from B to A.

In one example, the neural network between B 204 and client 1 (206) is the B portion 204 plus the A1 portion (206A), with the communication of data SB1 (206C) and SA1 (206B) completing the entire neural network. For this model, the training process is as follows: Server 202 creates A and B and sends their respective models A (206A, 208A, 210A) to each client 206, 208, 210. For all clients, the operation involves repeating the following in a linear or iterative manner across the group of clients until some condition occurs: Each client 206, 208, 210, in turn, downloads the latest model A from server 202 (note that this step differs between the approaches shown in Figures 2 and 3). Each client 206, 208, 210, in turn, performs a forward step on model A and sends the output of A (i.e., S only or activations at SA1 (206B), SA2 (208B), and SA1 (210B)) to server 202 along with the necessary labels. Server 202 performs a forward step on B using the SA received from each client 206, 208, 210. Server 202 computes the loss function, and server 202 performs backpropagation and computes gradients at the S layer. Server 202 sends only the gradients of S (i.e., SB1 (206C), SB2 (208C), and SB1 (210C)) to each client 206, 208, 210. This process is performed linearly across the different clients, such that operations occur first for client 206, then client 208, and then client 210. Clients 206, 208, and 210 perform backpropagation using the SB gradients received from server 202, and clients 206, 208, and 210 share the updated A (SA1 (206B), SA2 (208B), SAN (210B)) with server 202.

The horizontal axis in Figure 2 represents the time it takes for processing to be carried out from client to client in a round-robin fashion.

In one example, network A1 206A on client 1 may include a convolutional layer and an activation layer. Once it processes the data, client 1 (206) sends the results of its layer forward (SA1 (206B)) to the next layer in the network at server 202, which computes backpropagation, etc., as outlined above. The B network repeatedly processes different data from different clients 206, 208, 210 (in a round-robin fashion). Eventually, it arrives at an average reflection of the network. The network is not trained on all data from all clients 206, 208, 210 simultaneously. This has the advantage of processing the data faster and averaging B across the data as it is built. The final algorithm has not seen all the data. Model B cannot be tricked into revealing that data because it has never been trained on all the data.

Partitioned Learning in a Peer-to-Peer Environment Figure 3 illustrates a partitioned learning peer-to-peer approach. The model (neural network) is partitioned into two parts: one part (A) resides on the client side and includes an input layer, and the other part (B) resides on the server side and often includes an output layer. In Figure 3, the client-side part (A) is shown as A1 (306A) on client 306, A2 (308A) on client 308, and AN (310A) on client 310. The partitioned layer (S) refers to the layer in which A and B are partitioned. In Figure 3, SA represents the partitioned layer sent from A to B, and SB represents the partitioned layer sent from B to A.

In one example, the neural network between B and client 1 306 is part B plus part A1 306A, with the communication of data SB1 306C and SA1 306B completing the entire neural network. In this model, the training process is as follows: Server 302 creates A and B and sends A to clients 306, 308, and 310. For all clients, the process involves repeating the following until some condition occurs: First, the process involves downloading the latest A from the previous client.

Note that this step differs from the approach shown in other figures. The process then involves performing a forward step on A and sending the output of A (i.e., only the activations at S) along with the necessary labels to the server 302. The server 302 performs a forward step on B using the SA received from each client 306, 308, 310. The server 302 calculates the loss function, performs backpropagation, and calculates the gradient at S. The server 302 sends only the gradient of S (i.e., SB) to each client 306, 308, 310. The clients perform backpropagation using the SB gradients received from the server 302. The clients share their updated A with the server 302.

A peer-to-peer approach generally involves each client downloading model A directly from the most recently trained client, or more broadly, updated by a previously trained client. In this regard, the process of training clients can be performed in a round-robin fashion, with clients trained in turn. For example, if client 1 306 is trained first, in a peer-to-peer model, client 2 308 updates its client model A2 by downloading client-side model A1 from client 1 306, rather than client 2 308 updating client-side model A2 from server 302 or another trusted server. The previously trained model may be the most recently trained client model, or it may be another client's model previously trained based on some criteria. For example, client 1 306 and client 2 308 may train their respective models. Client 3 310 may need to update its client-side model and may implement an algorithm or process to determine which client-side model to download between client 1 306 and client 2 308. Note that the following disclosure implements a multi-model artificial intelligence training process applicable here. If client 1 306 processes images and its model A1 focuses on image processing, client 2 308 processes text and its model A2 focuses on text processing, and client 3 310 processes images, an algorithm or process would download client-side model A1 to client 3 310 as an update in a peer-to-peer environment.

In some scenarios, there is not enough information from the split learning to achieve proper training of the neural network. In this model, it is assumed that an appropriate training approach would be to aggregate A and B into plain text by simply stacking them (A and B) on server 302.

Federated and Partitioned Learning Figure 4 illustrates the training improvement to neural networks disclosed herein. This improvement can be characterized as a federated and partitioned learning approach, and addresses some of the deficiencies of the approaches disclosed above. Figure 4 introduces a parallel processing approach. Parallel and independent processing allows the model to train at a faster pace than the other models discussed above.

The federated partitioning learning approach does not perform the round-robin process described above. The server 402 partitions the network by "partition layers," which are user parameters inserted into the network definition code. The "top" of the network is kept on the server 402, and the "bottom" is sent to the respective data provider or client 406, 408, 410 (the terms client and data provider are used interchangeably herein). Training begins with the lowest network layer, which is the layer closest to the data. Each layer reads either the data (from the first layer) or the output of the previous layer (all other layers).

Layers can compute their outputs (hence the term "activations") based on any valid network architecture command (convolution, dropout, batch normalization, layer flattening, etc.) and activation function (relu, tanh, etc.). Once the last layer 406, 408, 410 on the data side computes its appropriate activations (i.e., outputs), those outputs are sent to the first layer on the "other side of the split," i.e., the first layer on the server side 402.

The next approach involves splitting the model as before. The model is split into two parts: (A) on the client side, containing the input layer, and (B) on the server side, often containing the output layer. (S) is the split layer. Clients or data providers 406, 408, 410 operate independently and send back answers, if any. Code on the server 402 processes the data and sends its output back to all clients equally as SBs (406C, 408C, 410C).

An example training process is as follows: Server 402 creates A and B and sends part A (406A, 408A, 410A) to clients 406, 408, 410. The following steps are repeated until a condition (e.g., accuracy) is met. All clients 406, 408, 410 simultaneously perform forward steps on A. Up until this point, all computations at clients 406, 408, 410 have been performed on independent servers, with no dependencies from one data server to another. This approach highlights one of the innovations disclosed herein: all these computations by clients/data providers 406, 408, 410 can all run simultaneously in parallel. This is in contrast to the linear or "round robin" approach described above.

Clients 406, 408, 410 each run portion A of the neural network (406A, 408A, 410A), generate a respective output for A (i.e., SA (406B, 408B, 410B)), and send the outputs to server 402. Server 402 receives three different "versions" of the activations (one from each of SA1, SA2, and SA3). At this point, server 402 processes those activations "appropriately." This may mean that server 402 takes different actions in different cases. For example, server 402 calculates a loss value for each client 406, 408, 410, and then server 402 calculates the average loss across all clients. Server 402 performs backpropagation using the average losses to calculate the gradient at S. The server 402 sends the gradient at S (i.e., SB(406C, 408C, 410C)) to all clients 406, 408, and 410.

In other words, training on the server side 402 proceeds much the same as described above. Once the first layer on the server side 402 is "complete" (by averaging or aggregating what is received from data providers 406, 408, 410), forward propagation occurs until the "top" of the network is reached. An additional innovation described in this disclosure is in the management of activations from data providers 406, 408, 410 and how they get averaged, aggregated, or otherwise processed. Once the system reaches the top of the model, the server 402 calculates the gradients needed for backpropagation and sends them back down through the partitioned network as shown in Figure 4.

As mentioned above, the processing and management of activations by server 402 can vary depending on a variety of factors. For example, consider the case where all three data providers 406, 408, 410 are supplying the same data (x-rays). In that case, the data is combined horizontally, which may conceptually mean that the data is one file "stacked" on top of other files. In this case, the activations that occur are more likely to be averaged out. The "average of each activation" is forwarded to the "top half" of the network.

In another case, data can be stacked "vertically" so that client 1 406 has the first 40 columns of data (e.g., blood tests), client 2 408 has the next 60 columns of data (e.g., electronic health record containing data such as age, weight, etc.), and client 3 410 has the last 100 columns of data (e.g., insurance information - previous claims, etc.). In this example, the three clients can be thought of as establishing a combined "record" of 200 columns (vertically aggregated across the page). In this case, activations are "vertically combined" and transferred to the server network. This and other approaches to combining data can be implemented. Note that the multimodal artificial intelligence model described in more detail below is based on the concept just described with respect to vertically combining activations. This concept is discussed in more detail below.

As mentioned above, clients 406, 408, and 410 operate in parallel in this embodiment. This reduces the time it takes to train the model, as all processing occurs in parallel. Furthermore, this data is delivered via a specific platform. The applications incorporated above provide examples of specific platforms that can be used to deliver the data disclosed herein, which are described in more detail below.

The global model in federated partition learning can be aggregated as follows: Once training is complete, the system aggregates a global model to be used for inference tasks using the following approach: In the first approach, the server selects one of the models Ai and merges it with its model B to form the global model. The selection of Ai can be done using one of the following methods: For example, random selection can be used, where the server randomly selects a model (Ai) for any client 406, 408, 410. This random selection may be influenced by other factors, such as the clients currently available online, the type of data each client processes (text data, image data, time data), or may be based on the transmission speed or network latency between the two entities. The server then stacks both parts Ai and B to generate the global model.

In another example, weighted client selection can be used. For this selection criterion, server 402 assigns each client a weight (i.e., a numerical value) that reflects importance based on their data, computing power, and other valuable assets they possess and contribute during the training process. For example, certain model sets (e.g., data in a particular language, data associated with a type of image, data associated with a patient set, data from a particular country or region, etc.) may be weighted more heavily in model development. Thus, if a country is selected, client devices in that country may be weighted more heavily than clients in other countries. For example, Japanese-based client devices may be used for 80% of the model data, while Australia may be 10% and Canada the remaining 10%. In another example, data from a particular clinic related to influenza or COVID outbreaks may be weighted more heavily. In yet another example, types of data may also be weighted more heavily. Image data may be used for 70% of the model, while text data may be 20% and time data may be 10%.

Yet another model selection may be based on accuracy. In this case, the server 402 may test the accuracy generated from each client model Ai and select the model that produces the "best" accuracy. A stakeholder may identify the "best" through a machine learning approach, for example. These are all models of the first approach.

A second approach could be to aggregate a global model by averaging all client models Ai{1,N}. Each client first encrypts its model using homomorphic encryption and then sends the encrypted Ai' data to the server 402. The server 402 adds all encrypted models, decrypts the results, and calculates their average. The averaged A is then stacked with B to generate the global model. One approach could be the default approach, or an optional approach could be provided. The decryption and averaging processes could also be distributed across different servers. For example, one process occurs on the client side and another process is executed by the server 402 to achieve the global model.

The approach may vary throughout the development of the model. For example, you can start training the model using the default approach, and then adjust the training to finish training the model using a weighting approach.

An example method is shown in FIG. 5 and may include splitting the neural network into a first portion and a second portion at the server (502), separately transmitting the second portion to the first client and the second client (504), and performing the following operations until a threshold is met:
(1) At the first client and the second client, simultaneously perform a forward step on the second portion to generate data SA1 and SA2.
(2) The first and second clients send SA1 and SA2 to the server.
(3) The server calculates the loss values of the first and second clients.
(4) At the server, the average loss across the first and second clients is calculated.
(5) On the server, perform backpropagation using the average loss to compute the gradient.
(6) The gradients are sent from the server to the first client and the second client (506).

Also covered are one or more computing devices that perform the above operations, and computer-readable storage devices that store instructions that, when executed, cause a processor to perform these operations. The operations may be performed in any order, and a method may include one or more operations.

In another aspect of the present disclosure, the platform described in the above-incorporated patent applications can provide the basis for exchanging data in any of the federated models. For example, each of the clients and/or servers may need to log on to the platform or one of the versions of the platform referenced in the applications incorporated herein. Thus, providing this functionality via a platform or exchange mechanism configured as disclosed in these applications is also covered as an aspect of the present disclosure.

Alternatively, the customer can select the SA and SB lines (vectors and numbers) that represent the weights that need to be propagated. If the client wants to lock down the data without the server knowing anything about it, the data can be homomorphically encrypted. The encryption process (which can include any encryption process) can be used with any of the approaches disclosed above.

The above-incorporated patent applications provide examples of platforms to which client devices and/or servers can or must log in to implement the federated partitioned learning approach disclosed herein.

Note that in one aspect, the steps disclosed herein can be performed by a "system." A system can include a server and one or more clients together, or can simply be functionally performed by a server. A system can also be a client or a group of clients, such as clients in a particular geographic area, or a group of clients in some manner performing the client-based functionality disclosed herein. In one aspect, a "server" can be a server-side computing device (physical or virtual) and a client-side computing device (physical or virtual). In one example, the server is on the client side and can receive the backpropagation output of each client-side model Ai and can synchronize the client-side global model across training rounds.

As such, each of the server-side system and the client-side system may perform any one or more of the operations disclosed herein. Claims may be included that recite steps occurring from the perspective of any apparatus disclosed herein. For example, steps of transmitting, computing, and receiving data may be claimed from the perspective of a server device, a client device, or a group of client devices, depending on the embodiment under consideration. All such communications from the perspective of an individual component or device may be included within the scope of a particular embodiment focused on that device.

In another aspect, the system may include a platform disclosed in the patent applications incorporated by reference and may perform steps in conjunction with the concepts disclosed above. Thus, a platform used to provide the federated partition learning process described herein is also an embodiment of the present disclosure, and steps may be recited relating to the use of that platform to train a model in a manner that maintains data privacy as described herein.

Typically, neural networks are trained on similar data types. For example, a neural network trained to receive images of patients or kidneys and identify cancer would be trained on images of cancerous and non-cancerous kidneys. Next, we describe a novel approach to training that uses different types of training data together to train a neural network using the federated split learning approach disclosed herein.

Multi-Model Artificial Intelligence Approach As mentioned above, MMAI's innovation is based on the idea of "vertical aggregation" described in the federated partitioning learning example. This example involves three clients 406, 408, and 410 all providing the same type of data: images (for stacking) or tabular data that are vertically joined. As the inventors explored the concept of vertical aggregation, they realized that it could be done with different types of data. For example, client 1 could provide images, client 2 could provide blood tests, and client 3 could provide doctor's text notes. The key difference is that all of these data types require different network architectures. In this case, the system developer cannot define one network and then have the server "partition" it. Therefore, part of the solution is to allow users to define the network "before partitioning" for each data provider and then define the network and aggregation techniques on the server. This approach is illustrated in Figures 6-10.

Figure 6 shows a multimodal artificial intelligence (MMAI) or machine learning (ML) platform 600. The MMAI approach reduces the computational requirements and communication overhead of other approaches. Furthermore, training speed is much faster, and the process maintains much higher data privacy, including the fact that the model remains private.

The MMAI platform 600 applies AI/ML techniques to multiple data types within one large AI model. Different types of data typically require different AI network architectures to achieve accurate results. For example, images typically require special filters (convolutions), while text and audio require different "time-series-like" processing, and tabular data often works best with ML or forward-propagation architectures. The problem is that images are best understood by looking at all the pixels together and "convolving" them in various ways, whereas audio is best understood in the context of what comes before and after a particular sound (i.e., in a manner similar to time-series data), etc. Due to these processing differences, today's "state-of-the-art" systems typically process one data type (image, text, audio, tabular, etc.).

Most AI researchers recognize that adding more unique data to a model can dramatically improve its "next generation" accuracy. This is essentially the same as providing the model with more data to give it more context for discovering interesting differences between cases. An example of this concept is a model that diagnoses atrial fibrillation (A-fib) by examining ECG (electrocardiogram) data. This model can reach a certain level of accuracy based on the ECG data alone, but when researchers add age, gender, height, and weight to the ECG data, the model becomes much more accurate. The improvement in accuracy occurs because the addition of four data types gives the model a better understanding of what "equivalent" ECGs look like. Adding four items or characteristics of data allows the data to be more granular.

The MMAI platform 600, shown in Figure 6, introduces a new generation of encryption toolset to improve the training and protection of private data. The MMAI platform 600 augments the data by providing the model with more data than is typically used to train AI/ML models. This approach adds large amounts of data by combining various data types (e.g., images and tabular data).

Figure 6 shows a first external data source 602, shown as Wells Fargo Bank. Wells Fargo data 602a is encrypted (602b), and a package of encrypted data 602c is sent to a private AI infrastructure 603. A second external source of data 604 is shown as Citibank. Citibank data 604a is encrypted (604b), and a package of encrypted data 604c is sent to a private AI infrastructure 603. A third external source of data 606 is shown as coming from Bank of America. Bank of America data 606a is encrypted (606b), and a package of encrypted data 606c is sent to a private AI infrastructure 603. The AI infrastructure 603 includes a first module 608 that privately examines, selects, and pre-processes all data 610 from different sources 602, 604, 606. In this example, all of the sources are identified as banks, but the data may be structured differently and the respective data may be different. Of course, not all of the external sources of data 602, 604, 606 need to be of the same type, i.e., banks. The use of banks is an example. The external sources 602, 604, 606 could be, for example, hospitals, clinics, universities, etc. The basic concept is that the data types from the various different external sources 602, 604, 606 may be different.

The private AI infrastructure 603 may include components that privately search for, select, and preprocess relevant features from all of the data 602c, 604c, and 606c received for training. Features 612 represent a subset of data 610 that may result from processing by components in the private AI infrastructure 603. In operations 614 and 616, the AI infrastructure 603 privately trains new deep and statistical models on the selected data 612, and in operation 618 predicts any private and sensitive data, which may include images, video, text, and/or other data types. The AI infrastructure 603 may then sell or license access to the presented new models in operation 620.

Figure 7 shows another variation of the split learning technique 700. This approach uses a blind correlation process for training on different types of data, resulting in lower computational requirements and communication overhead, and improved model training. Building on the A-fib model example above, another source of even more data for the model would be to include a chest x-ray for each case considered by the model. Unfortunately, the typical processing of x-ray images is inconsistent with the typical processing of tabular ECG data. With some minor technical additions, the split federated learning tool disclosed above can be used to address this incompatibility issue; new instructions can be provided to the tool to enable it to process different data types within the existing pipeline.

In this case, rather than an "automatic" split of the network architecture, a variation on this idea allows the network architect (i.e., the data scientist developing the algorithm) to specify the specific network components needed for each data type. Each data type requires network architecture layers relevant to that data type (i.e., a convolutional layer for images, a recurrent/long short-term memory layer for audio, a forward propagation layer for tabular data, etc.). These different layers, each specific to the data type in question, are specified to run on the "data server" side (much like an independent network within itself). The last layer of each "independent network" (per data type) sends its activations "across the split" to the "server side." On the algorithm server side, there is one coherent "network" that processes the incoming activations (from the data server side) appropriately. In some ways, this approach is similar to an "ensemble of networks" (on the data server side) being aggregated into one final network on the algorithm server side (which ultimately produces the final "answer" from the "ensemble" of networks).

As previously mentioned, split learning is a collaborative deep learning technique in which a deep learning network or neural network (NN) can be split into two parts: a client-side network A and a server-side network B. The NN includes weights, biases, and hyperparameters. In Figure 7, clients 702, 704, and 706, where data resides, commit only to the client-side part of the network, and server 710 commits only to the server-side part of the network. The client-side and server-side parts collectively form the complete network NN.

Network training is accomplished through a series of distributed training processes. Forward and backpropagation can occur as follows: Using raw data, a client (e.g., client 702) trains a client-side network 702A up to a particular layer of the network, which may be called a cut layer or partition layer, and sends the cut layer activations to the server 710. The server 710 uses the activations received from client 702 to train the remaining layers of the NN. This completes a single forward propagation step. A similar process occurs in parallel for a second client 704 and its client-side network 704A, and its data and generated activations, which are sent to the server 710. A further similar process occurs in parallel for a third client 706 and its client-side network 706A, and its data and generated activations, which are sent to the server 710.

The server 710 then performs backpropagation up to the cut layer and sends the activation gradients to each of the clients 702, 704, and 706. Using the gradients, each client 702, 704, and 706 then performs backpropagation over the remaining network 702A, 704A, and 706A. This completes a single pass of backpropagation between the clients 702, 704, and 706 and the server 710.

This process of forward and backpropagation continues until the network has been trained on all available clients 702, 704, 706 and has reached convergence. In partitioned learning, the architecture configuration is assumed to be performed by a trusted party with direct access to the main server 710. This authorized party selects the ML model (based on the application) and the network partition (to find the cut layer) at the start of learning.

As mentioned above, the concepts introduced in this disclosure pertain to clients 702, 704, and 706, each providing different types of data, but also when the different types of data have common associations. Thus, the selection of a machine learning model can be based on the type of data being processed on the client side, and the process of finding a cut layer can also depend on the type of data or the imbalance of different types of data. For example, for widely different data types across clients 702, 704, and 706, the cut layer can be selected to have more or fewer layers on the client-side network 702A, 704A, and 706A. In another aspect, the number of layers before the cut layer or segmentation layer can vary by client. Client 702 may be processing images and require eight layers before the cut layer, while client 704 may be processing text and require only four layers before the cut layer. In this regard, there is no requirement that the number of layers on the client-side networks 702A, 704A, 706A be the same, as long as the cut layer vectors, activations, or activation layers are consistent across different clients 702, 704, 706 with different types of data.

Synchronization of the learning process with multiple clients 702, 704, 706 can be done in either centralized or peer-to-peer mode. In centralized mode, before initiating training on the server 710, clients 702, 704, 706 update their client-side models 702A, 704A, 706A by downloading model parameters from a trusted third-party server 710 that maintains updated client-side models uploaded by the most recently trained clients. In peer-to-peer mode, clients 702, 704, 706 update their client-side models by downloading directly from the most recently trained clients. As previously mentioned, the previously trained model may be similar in data type to the current client whose model needs to be updated. For example, similarity may be based on image, text data, audio data, video data, time data, etc. Thus, there may be intelligent selection of which previously trained client model to use for download from a peer. Processing by the server 710 may also be split, in some cases, between some processing on the server side and other processing on the client-side federated server.

As introduced above, client 1 702, client 2 704, and client 3 706 may have different data types. The server 710 creates two portions of the network and sends one portion 702A, 704A, and 706A to all clients 702, 704, and 706. The system repeats certain steps until accuracy or other conditions are met, such as all clients sending their data to their portion of the network, and sends the output to the server 710. The server 710 calculates a loss value for each client and an average loss across all clients. The server 710 can update its model using a weighted average of the gradients calculated during backpropagation and send the gradients back to all clients 702, 704, and 706. The clients 702, 704, 706 receive the gradients from the server 710, and each client 702, 704, 706 performs backpropagation on the client-side networks 702A, 704A, 706A, calculating a respective gradient for each client-side network 702A, 704A, 706A. The respective gradients from the client-side networks 702A, 704A, 706A can then be sent back to the server 710, which averages the client-side updates and sends a global result back to all clients 702, 704, 706.

Note that the functionality of server 710 could also be split across several servers, each performing different operations (e.g., one server updating the model and other servers each located in a different area averaging updates for local clients). In the case of Figure 7, clients 702, 704, and 706 all process different types of data that may or may not typically be processed to develop AI models.

For illustrative purposes, the process can be described using the A-fib model described above. Client 1 702 can have ECG data, Client 2 704 can have X-ray data, and Client 3 706 can have genetic data. As shown in FIG. 6, for example, Client 1 702 could be a hospital, Client 2 704 could be a medical diagnostic imaging company, and Client 3 706 could be a bank or financial institution. One of the clients could also have time-based data, such as progressive information about a patient compared to visits to the hospital for regular checkups.

The approach shown in Figure 7 illustrates how the system can implement new user instructions that allow the user to combine different data types with the "right" processing before splitting or cutting layers, or as shown in the blind decorrelation block 708. Each of these parts of the model can be independent and operate independently. In one aspect, the processing performed by the blind correlation block 708 results in an activation layer or activations that are forwarded to the server 710. This approach is similar to the approach described above with the addition of differences in data types between clients 702, 704, and 706.

The server 710 combines the activation layers in one of a number of ways. The server 710 can average them (also described above) or concatenate them into one long activation layer. In another aspect, the server 710 can apply any mathematical function to achieve the desired combination of activation layers. The server 710 can then further process the combined activation layers using any suitable network architecture. In one aspect, a client-side server can receive the gradients, average the gradients to generate a global model for the various clients 702, 704, 706, and send the global model to the server 710 for concatenation or further processing.

The ideas shown in Figures 6 and 7 represent an extension and application of the Partitioned Federated Learning toolset, providing a platform of off-the-shelf tools for combining different data types into superset AI models. All processing can be done privately, and offerings can also be included in the marketplace, as described in the above-referenced patent applications.

The system can not only combine different data types, but also different AI/ML techniques. For example, client 1 702 can be a CNN (convolutional neural network), client 2 704 can be an ML routine (i.e., XGBoost), and client 3 706 can apply a different technique. In this respect, different AI/ML techniques differ, but as long as the resulting data at the cut layer is consistent and properly structured, forward and back propagation can be performed to train the model.

To help those skilled in the art understand how the MMAI approach works, below is an example of actual commands for each data type coming from the three data providers 702, 704, 706. This code uses Python numbering conventions, so it starts with builder 0 (tabular data from data provider 1 702). Builder 1 in this example is for CT scan or image data. Commands are similar for X-ray, MRI, and/or other images. Builder 2 (from data provider 704) is text data. Note the "lstm" command, which stands for "long/short term memory." The "server" builder command defines a network that aggregates the other three at the "top" on the other side of the split.
builder0=tb. NetworkBuilder()
builder0. add_dense_layer(100,120)
builder0. add_relu()
builder0. add_dense_layer(120,160)
builder0. add_relu()
builder0. add_dropout(0.25)
builder0. add_dense_layer(160,200)
builder0. add_relu()
builder0. add_split()
builder1=tb. NetworkBuilder()
builder1. add_conv2d_layer(1, 32, 3, 1)
builder1. add_batchnorm2d(32)
builder1. add_relu()
builder1. add_max_pool2d_layer(2,2)
builder1. add_conv2d_layer(32,64,3,1)
builder1. add_batchnorm2d(64)
builder1. add_relu()
builder1. add_max_pool2d_layer(2,2)
builder1. add_flatten_layer()
builder1. add_split()
builder2=tb. NetworkBuilder()
builder2. add_lstm_layer(39,100, batch_first=True)
builder2. add_dense_layer(100,39)
builder2. add_split()
server_builder=tb. NetworkBuilder()
server_builder. add_dense_layer(60000,8000),
server_builder. add_relu()
server_builder. add_dense_layer(8000,1000),
server_builder. add_relu()
server_builder. add_dense_layer(1000,128),
server_builder. add_relu()
server_builder. add_dense_layer(128,1)

FIG. 8 illustrates an exemplary method 800 for providing an MMAI concept from a client's perspective. The method includes receiving a first data set having a first data type from a first data source (802), training a first client-side network with the first data set and generating first activations (804), receiving a second data set having a second data type from a second data source (806), and training a second client-side network with the second data set and generating second activations (808).

The method may further include sending the first activations and the second activations to a server-side network (810), the server-side network being trained based on the first activations and the second activations to generate gradients, and receiving the gradients at the first client-side network and the second client-side network (812). The first data type and the second data type may be different data types, such as one being image-based and the other being text- or time-based, such as audio.

9 illustrates an exemplary method 900 from the perspective of both the server 710 and one or more clients 702, 704, 706. The method may include dividing the neural network into a first client-side network, a second client-side network, and a server-side network (902); transmitting the first client-side network to a first client, where the first client-side network is configured to process first data from the first client, the first data having a first type, and the first client-side network may include at least one first client-side layer (904); and transmitting the second client-side network to a second client, where the second client-side network is configured to process second data from the second client, the second data having a second type, and the second client-side network may include at least one second client-side layer, the first type and the second type having a common association (906).

The method may further include training a first client-side network with first data from a first client to generate first activations (908), transmitting the first activations from the first client-side network to a server-side network (910), training a second client-side network with second data from a second client to generate second activations (912), transmitting the second activations from the second client-side network to the server-side network (914), training at least one server-side layer of the server-side network based on the first activations and the second activations to generate gradients (916), and transmitting the gradients from the server-side network to the first client-side network and the second client-side network (918).

The common association between different types of data can include at least one of a device, a person, a consumer, a patient, a business, a concept, a medical condition, a group of people, a process, a product, and/or a service. Any concept, device, or person can be a common association or theme among various different types of data obtained from various clients and processed up to the cutting or splitting layer by various independent client-side networks. The server-side network can include a global machine learning model. The neural network can include weights, biases, and hyperparameters. Hyperparameters typically relate to parameters whose values are used to control the learning process, such as topology parameters or the size of the neural network. For example, the learning rate, mini-batch size, number of client-side layers, or parameters related to controlling processes that affect or relate to various data types can represent hyperparameters.

The at least one first client-side layer and the at least one second client-side layer can each include the same number of layers or different numbers of layers. Because they operate independently, the client-side networks can have different numbers of layers as long as they process data and generate vectors or activations in the appropriate format to pass to the server-side network for further training. There may be cut layers between the server-side network and the first and second client-side networks.

10 illustrates an exemplary method 1000 from the perspective of the server 710. The method may include dividing the neural network into a first client-side network, a second client-side network, and a server-side network (1002); transmitting the first client-side network to a first client, where the first client-side network is configured to process first data from the first client, the first data having a first type, and the first client-side network may include at least one first client-side layer (1004); and transmitting the second client-side network to a second client, where the second client-side network is configured to process second data from the second client, the second data having a second type, and the second client-side network may include at least one second client-side layer, and the first type and the second type have a common association (1006).

The method may further include receiving, at the server-side network, first activations from training a first client-side network on first data from a first client (1008); receiving, at the server-side network, second activations from training a second client-side network on second data from a second client (1010); training at least one server-side layer of the server-side network based on the first activations and the second activations to generate gradients (1012); and transmitting the gradients from the server-side network to the first client-side network and the second client-side network (1014).

Note that in each case, some of the server 710's processes related to training may be performed by the server 710, while other parts, such as averaging values across various clients, may be performed by a different server (not shown), which may be at the client site, at another location, or across different clients.

This approach allows the federated partition learning toolset to be used in a new way: as the system partitions the neural network, with blind correlation 708, the system can take the resulting trained model and partition it to make it more difficult to apply training inference attacks. Because the system can partition the neural network in half (or into two parts) in the manner described above, all that is exchanged from neural network parts 702A, 704A, and 706A is a string or sequence of numbers, also described as activation layer numbers. Because these are simply sequences of numbers or letters, what happens in the first neural network part 702A can be different from what happens in the second neural network part 704A. For example, the first neural network part 702A may be two layers deep, while the second neural network part 704A may be 90 layers deep. As long as each output resolves to a properly configured sequence of numbers to send to the top of the neural network 710, forward and back propagation can work and training can be accomplished. This understanding paves the way for the novel concept disclosed herein, that different types of data to be processed by different portions 702A, 704A, 706A of the neural network can be received and processed appropriately to train the model. If the system can create different bottom halves 702A, 704A, 706A for different clients, the clients 702, 704, 706 do not need to generate or process the same types of data (e.g., between text and images), and the appropriately formatted neural network portions 702A, 704A, 706A can process that heterogeneous data and generate structured output that can be sent to the server 710.

In one example, client 1 702 can provide a person's ECG, client 2 704 can provide a cardiac chest X-ray, and client 3 706 can provide a genetic profile of the four proteins of most interest in the patient's blood. If the neural network portions 702A, 704A, 706A can process each different type of data into the correct vector structure for output and provide the different types of data to server 710, server 710 can configure an appropriate neural network to combine all of that information and train a model that can be used to make a diagnosis that can utilize the different types of data.

In one aspect, although neural network portions 702A, 704A, and 706A each process different types of data, there are some correlation factors associated with the data. In the example above, all of the data may generally relate to the same person; some data may be ECG-related and other data may be associated with genetic profiles, but they are all still from the same person. Thus, one aspect of the present disclosure is that the data have common associations. In another aspect, the data may not relate to the same person, but the common association may relate to age, gender, race, a project, concept, climate, stock market, or other factors. For example, all of the data may relate to women aged 30-35. Thus, the common associations have some flexibility in how they can be applied.

In another example, the data could be images from a camera of a jet engine stream, another data stream could be sensor data, other data could be the flight characteristics of an airplane, and the common association could be the airplane. In another aspect, the common association could be one type of data being purchasing habits, another type of data being web surfing patterns, another type of data being emails sent by a user, another type of data being audio from Siri or other voice processing tools, another type of data being physical stores frequented by consumers, the user's current location, etc. The output of the server could be advertisements served to the user based on an analysis of the different types of input. Thus, the common association can relate to any available concept where different types of data can be related to the concept.

FIG. 11 illustrates an example of a computing device that can be connected to and used with any of the systems disclosed herein. In this example, FIG. 11 illustrates a computing system 1100 including components that communicate electrically with each other using a connection 1105, such as a bus. The system 1100 includes a processing unit (CPU or processor) 1110 and system connections 1105 that couple various system components to the processor 1110, including system memory 1115, such as read-only memory (ROM) 1120 and random access memory (RAM) 1125. The system 1100 can include a cache of high-speed memory directly connected to, in close proximity to, or integrated as part of the processor 1110. The system 1100 can copy data from the memory 1115 and/or storage device 1130 to the cache 1112 for quick access by the processor 1110. In this manner, the cache can provide performance improvements that avoid delays to the processor 1110 while waiting for data. These and other modules can control, or be configured to control, the processor 1110 to perform various actions. Other system memory 1115 may be used as well. Memory 1115 may include multiple different types of memory with different performance characteristics. Processor 1110 may include any general-purpose processor, hardware or software services or modules, such as service (module) 1 1132, service (module) 2 1134, and service (module) 3 1136 stored on storage device 1130, configured to control processor 1110, as well as special-purpose processors in which software instructions are embedded in the actual processor design. Processor 1110 may be a completely self-contained computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

To enable user interaction with device 1100, input device 1145 can represent any number of input mechanisms, such as a microphone for audio, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. Output device 1135 may be one or more of numerous output mechanisms known to those skilled in the art. In some examples, a multimodal system allows a user to provide multiple types of input to communicate with device 1100. Communications interface 1140 generally can orchestrate and manage user input and system output. There is no limitation to operation with a particular hardware configuration, and the basic functionality herein can be easily replaced with improved hardware or firmware configurations as they are developed.

Storage device 1130 is non-volatile memory and may be a hard disk or other type of computer-readable medium capable of storing data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cartridge, random access memory (RAM) 1125, read-only memory (ROM) 1120, and hybrids thereof.

Storage device 1130 may include services or modules 1132, 1134, 1136 for controlling processor 1110. Other hardware or software modules are also contemplated. Storage device 1130 may be connected to system connection 1105. In one aspect, hardware modules that perform specific functions may include software components stored on computer-readable media in association with the necessary hardware components, such as processor 1110, connection 1105, output device 1135, etc., to perform the functions.

In some cases, such a computing device or apparatus may include a processor, microprocessor, microcomputer, or other component of a device configured to perform the steps of the methods disclosed above. In some examples, such a computing device or apparatus may include one or more antennas for transmitting and receiving RF signals. In some examples, such a computing device or apparatus may include an antenna and modem for transmitting, receiving, modulating, and demodulating RF signals, as described above.

Components of a computing device may be implemented in circuitry. For example, components may include and/or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and/or other suitable electronic circuitry), and/or may include and/or be implemented using computer software, firmware, or any combination thereof, to perform the various operations described herein. A computing device may further include a display (as an example of an output device, or in addition to an output device), a network interface configured to communicate and/or receive data, any combination thereof, and/or other component(s). The network interface may be configured to communicate and/or receive Internet Protocol (IP)-based data or other types of data.

The methods described above are illustrated as logical flow diagrams, where the operations represent sequences of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and/or in parallel to implement a process.

Furthermore, the methods disclosed herein may be executed under the control of one or more computer systems configured with executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) collectively executed on one or more processors, by hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instruction(s) and/or data. Computer-readable media can also include non-transitory media capable of storing data and that do not involve carrier waves and/or transitory electronic signals propagated via wireless or wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, and memory devices. Computer-readable media may store code and/or machine-executable instructions, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment can be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bitstreams, etc. However, when mentioned, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art will understand that embodiments may be practiced without these specific details. For clarity of explanation, in some instances, the technology may be presented as including individual functional blocks, including devices, device components, steps, or routines in methods embodied in software, or a combination of hardware and software. Additional components other than those shown in the figures and/or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

Particular embodiments may be described above as a process or method that is depicted as a flowchart, flow diagram, data flow diagram, structure diagram, or block diagram. While a flowchart may describe operations as sequential, many operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but may include additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

The processes and methods according to the above-described examples may be implemented using computer-executable instructions stored on or available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, special-purpose computer, or processing device to perform a particular function or group of functions. Some of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, intermediate format instructions such as binary, assembly language, firmware, or source code. Examples of computer-readable media that may be used to store instructions, information used, and/or information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., a computer program product) to perform the necessary tasks may be stored on a computer-readable or machine-readable medium. A processor(s) may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein may also be embodied in a peripheral device or an add-in card. As a further example, such functionality may be implemented on a circuit board among different chips or different processes executed on a single device.

The instructions, media for communicating such instructions, computing resources for executing them, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

While the foregoing description describes aspects of the present application with reference to specific embodiments thereof, those skilled in the art will recognize that the present application is not limited thereto. Accordingly, while exemplary embodiments of the present application have been described in detail herein, it should be understood that the inventive concepts may be otherwise embodied and used in various ways, and that the appended claims are intended to be construed to include such variations except as limited by the prior art. Various features and aspects of the above-described application may be used individually or together. Moreover, the embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the present specification. Accordingly, the specification and drawings are to be regarded as illustrative, and not restrictive. For purposes of illustration, methods have been described in a particular order. It should be understood that in alternative embodiments, methods may be performed in an order different from that described.

Those skilled in the art will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of this description.

When a component is described as being "configured" to perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

The term "coupled" refers to any component that is directly or indirectly physically connected to another component and/or that is in direct or indirect communication with another component (e.g., connected to the other component via a wired or wireless connection and/or other suitable communication interface).

Claim language or other language reciting "at least one" of a set and/or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. As another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A, B, and C. The language "at least one" of a set and/or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can further include items not recited in the set of A and B.

While various examples and other information have been used to describe aspects within the scope of the appended claims, those skilled in the art may use these examples to derive various implementations, and therefore no limitation of the claims should be implied based on the specific features or arrangements of such examples. Furthermore, while some subject matter may have been described in language specific to example structural features and/or method steps, it should be understood that the subject matter defined in the appended claims is not necessarily limited to these described features or acts. For example, such functionality may be distributed or performed in a different manner by components other than those identified herein. Rather, the described features and steps are disclosed as example components of systems and methods within the scope of the appended claims.

Claim language reciting "at least one" of a set indicates that one member of the set or multiple members of the set satisfy the claim. For example, claim language reciting "at least one of A and B" means A, B, or A and B.

Claims (20)

a processor of the server dividing the neural network into a first client-side network, a second client-side network, and a server-side network;
a processor of the server transmitting the first client-side network to a first client, wherein a processor of the first client is configured to process first data from the first client via the first client-side network, the first data having a first type, and the first client-side network including at least one first client-side layer;
a processor of the server transmitting the second client-side network to a second client, wherein a processor of the second client is configured to process second data from the second client via the second client-side network, the second data having a second type, the second client-side network including at least one second client-side layer, and the first type and the second type being different types but having a common association;
a processor of the first client training the first client-side network with first data from the first client to generate a first activation;
a processor of the first client transmitting the first activation from the first client-side network to the server-side network;
a processor of the second client training the second client-side network with second data from the second client to generate second activations;
a processor of the second client transmitting the second activation from the second client-side network to the server-side network;
a processor of the server training at least one server-side layer of the server-side network based on the first activations and the second activations to generate gradients;
a processor of the server transmitting the gradients from the server-side network to the first client-side network and the second client-side network;
A method comprising: The method of claim 1, wherein the common association includes at least one of a device, a person, a consumer, a patient, a business, a concept, a medical condition, a group of people, a process, a product, and/or a service. The method of claim 1, wherein the server-side network includes a global machine learning model. The method of claim 1, wherein the neural network includes weights, biases, and hyperparameters. The method of claim 1, wherein the at least one first client-side layer and the at least one second client-side layer include the same number of layers or different numbers of layers. The method of claim 1, wherein a cut layer exists between the server-side network and the first and second client-side networks. The method of claim 1, wherein the first type includes text data and the second type includes image data. The method of claim 7, wherein the first client-side network and the second client-side network are independent and operate independently. The method of claim 1, wherein the first type includes tabular data and the second type includes image data. 1. A system comprising:
a processor;
10. A computer-readable storage device that, when executed by the processor, causes the processor to:
Dividing the neural network into a first client-side network, a second client-side network, and a server-side network;
transmitting the first client-side network to a first client, the first client-side network configured to process first data from the first client, the first data having a first type, and the first client-side network including at least one first client-side layer;
transmitting the second client-side network to a second client, the second client-side network configured to process second data from the second client, the second data having a second type, the second client-side network including at least one second client-side layer, the first type and the second type being different types but having a common association;
receiving, at the server-side network, a first activation from training the first client-side network with first data from the first client;
receiving, at the server-side network, second activations from training the second client-side network with second data from the second client;
training at least one server-side layer of the server-side network based on the first activations and the second activations to generate gradients;
transmitting the gradient from the server-side network to the first client-side network and the second client-side network;
a computer-readable storage device storing instructions for performing operations, including:
Including, the system. The system of claim 10, wherein the common association includes at least one of a device, a person, a consumer, a patient, a business, a concept, a medical condition, a group of people, a process, a product, and/or a service. The system of claim 10, wherein the server-side network includes a global machine learning model. The system of claim 10, wherein the neural network includes weights, biases, and hyperparameters. The system of claim 10, wherein the at least one first client-side layer and the at least one second client-side layer include the same number of layers or different numbers of layers. The system of claim 10, wherein a cut layer exists between the server-side network and the first and second client-side networks. The system of claim 10, wherein the first type and the second type are different types of data. The system of claim 10, wherein the first type includes tabular data or time series data, and the second type includes image data. a processor of the server dividing the neural network into a first client-side network, a second client-side network, and a server-side network;
a processor of the server transmitting the first client-side network to a first client, wherein a processor of the first client is configured to process first data from the first client via the first client-side network, the first data having a first type, and the first client-side network including at least one first client-side layer;
a processor of the server transmitting the second client-side network to a second client, wherein a processor of the second client is configured to process second data from the second client via the second client-side network, the second data having a second type, the second client-side network including at least one second client-side layer, and the first type and the second type being different types but having a common association;
a processor of the server receiving, at the server-side network, a first activation from training the first client-side network with first data from the first client;
a processor of the server receiving, at the server-side network, second activations from training the second client-side network with second data from the second client;
a processor of the server training at least one server-side layer of the server-side network based on the first activations and the second activations to generate gradients;
a processor of the server transmitting the gradients from the server-side network to the first client-side network and the second client-side network;
A method comprising: The method of claim 18, wherein the first type and the second type are different types of data. The system of claim 10 , wherein the first type comprises tabular or time series data and the second type comprises image data.