JP7514716B2 - Information processing system, hierarchical edge system, and information processing method - Google Patents

Description

The present invention relates to a technology for processing data collected using communication means.

As communication technology and sensors become faster and more functional, it is now possible to connect entire wide-area social systems, spanning people and things such as automobiles and robots. In this IoT (Internet of Things) environment, the concept of digital twins has been proposed, which sends information from the physical world to cyberspace and recreates the physical world environment within cyberspace.

By utilizing digital twins, it is expected that not only will it be possible to monitor the physical space, but it will also be possible to realize a system in which the entire system is automatically linked based on real-time predictions in cyberspace. In other words, it will be possible to predict and respond to changes in the real world based on these predictions in cyberspace.

Such systems will be useful for controlling entire social systems in real time with low environmental impact, for example, in smart factories where robots and humans work together, in linking autonomous trains and automobiles, and in controlling the supply of electricity to moving objects (robots, mobility, etc.).

The fifth-generation mobile communication system, known as 5G, is scheduled to begin service in Japan in 2020. Network slicing and edge computing are attracting attention as 5G technologies. Network slicing multiplexes virtual, independent logical networks according to application on the same physical network architecture. Edge computing distributes processing devices (MEC (Mobile Edge Computing, or Multi-access Edge Computing) servers) to locations (edges) physically close to users and terminals, and processes data at the edge of the network.

In addition, computational methods include artificial intelligence (AI) consisting of deep neural networks (DNN) and reservoir computing, as well as annealing machines. A reservoir computer is a type of recurrent neural network (RNN) that consists of three layers: an input layer, an intermediate layer (reservoir layer), and an output layer (read-out neuron layer), and learns, for example, through supervised learning. An annealing machine is a processing device that applies the Ising model, etc. to solve optimization problems.

Regarding network slicing, see Patent Document 1. Also, as an example of an annealing machine, see Patent Document 2.

JP 2020-136788 A JP 2018-206127 A

In order to connect the entire wide-area social system and control it without delay, data collection, data processing, and system control must occur in real time.

For this purpose, artificial intelligence (AI) consisting of DNN, reservoir computers, etc., or processing using annealing machines such as those described in Patent Document 2 have been proposed, but with the increase in data volume and demand for real-time processing, it is necessary to expand processing capacity or improve functionality.

The object of the present invention is to provide a data processing technology that can be adapted according to the required amount of data processing and the processing content.

One aspect of the present invention is an information processing system that includes a processing unit that includes at least one processing device selected from a function approximator and an annealing machine, and a data flow controller that receives data transmitted from a sensor side and transmits the data to the processing unit, and the data flow controller rearranges the data to be transmitted to the processing unit based on the transmission timing of the sensor side.

Another aspect of the present invention is a hierarchical edge system that includes multiple information processing systems having the above-described configuration, and has a hierarchical structure of a first information processing system, a second information processing system, and a third information processing system, in that order from the sensor side.

Another aspect of the present invention is an information processing method for processing data transmitted from a sensor side by a processing unit including at least one processing device selected from a function approximator and an annealing machine. A data flow controller is used to receive data transmitted from the sensor side and transmit it to the processing unit, and the data transmitted from the sensor side includes at least one of data obtained from different sensors, data that has passed through different routes, and data by different communication means, and the data flow controller rearranges the data to be transmitted to the processing unit based on the transmission timing of the sensor side.

We can provide data processing technology that can be adapted to the amount of data processing required and the processing content.

FIG. 1 is a block diagram showing a basic configuration of an information processing system according to a first embodiment. FIG. 11 is a block diagram showing the overall configuration of an information processing system according to a second embodiment. FIG. 11 is a table showing a list of processing patterns possible in the information processing system according to the second embodiment. FIG. 11 is a block diagram illustrating an example of circuit implementation of an information processing system according to a third embodiment. FIG. 13 is a block diagram illustrating an example of the configuration for feature extraction according to a fourth embodiment. FIG. 13 is a block diagram explaining the concept of a system in which a plurality of annealing machines perform processing in cooperation with each other according to a fifth embodiment. FIG. 13 is an internal block diagram of an annealing machine according to a fifth embodiment. 13 is an operation time chart of the annealing machine of Example 5. FIG. 13 is a conceptual diagram showing the concept of connection of adjacent spins in the annealing machine of Example 5. FIG. 23 is a block diagram explaining the concept of a system for solving the same problem using multiple annealing machines according to a sixth embodiment. FIG. 13 is a block diagram showing an example of a system for scaling up reservoir computing in a wireless manner according to the seventh embodiment. FIG. 13 is an internal block diagram of the reservoir of Example 7. FIG. 23 is a table showing the functions shared by the server and reservoir, as well as the data to be transmitted and the destination in the system of Example 7. 13 is an operation time chart of the reservoir of the seventh embodiment. FIG. 13 is a block diagram illustrating the concept of a system for performing the same process using multiple reservoirs in Example 8. A table showing the functions shared by the server and reservoir, as well as the data to be transmitted and the destination in the system of Example 8. 13 is an operation time chart of the reservoir of Example 8. FIG. 13 is a block diagram showing an example of a system for expanding the scale of a DNN in a ninth embodiment using wireless connections. A table showing the functions shared by the server and reservoir, as well as the data to be transmitted and the destination in the system of Example 9. FIG. 23 is a conceptual diagram showing an example of simultaneously processing multiple data in real time by the prediction planning unit according to the twelfth embodiment. FIG. 23 is a block diagram showing an example of edges having a hierarchical structure according to the thirteenth embodiment. FIG. 23 is a block diagram showing details of an edge according to a thirteenth embodiment. Block diagram showing the concept of staged composite AI. A block diagram showing the concept of concurrently analyzing the current situation and predicting the future.

The following describes the embodiments in detail with reference to the drawings. However, the present invention should not be interpreted as being limited to the description of the embodiments shown below. It will be easily understood by those skilled in the art that the specific configuration can be changed without departing from the concept or spirit of the present invention.

In the configuration of the invention described below, the same parts or parts having similar functions are designated by the same reference numerals in different drawings, and duplicate descriptions may be omitted.

When there are multiple elements with the same or similar functions, they may be described using the same reference numerals with different subscripts. However, when there is no need to distinguish between multiple elements, the subscripts may be omitted.

The designations "first," "second," "third," and the like in this specification are used to identify components and do not necessarily limit the number, order, or content. Furthermore, numbers for identifying components are used in different contexts, and a number used in one context does not necessarily indicate the same configuration in another context. Furthermore, this does not prevent a component identified by a certain number from also serving the function of a component identified by another number.

The position, size, shape, range, etc. of each component shown in the drawings, etc. may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings, etc.

The publications, patents and patent applications cited herein are incorporated by reference in their entirety into the present specification.

<Basic configuration of information processing system>
1 is a block diagram showing the basic configuration of an information processing system according to embodiment 1. This system includes a prediction and planning unit 100, a sensor 200, and an output unit 300.

The prediction and planning unit 100 includes a deep neural network (DNN) 101, a reservoir 102, an annealing machine 103, and a server 104. The prediction and planning unit 100 receives data from a sensor 200 as input, and outputs the calculation results to an output unit 300. The calculation results of the prediction and planning unit 100 are sent to the output unit 300, for example, by wireless communication. The calculation results of the prediction and planning unit 100 are, for example, recognition, understanding, judgment, prediction, or action planning (these are sometimes collectively referred to as inference. The processing and output of the DNN 101, reservoir 102, and annealing machine 103 themselves are sometimes referred to as inference.)

The sensor 200 collects data on the physical space, such as the position and movement of people, the state of machines and robots, and the state of transportation or the environment, using known sensor devices and cameras, and sends the data to the prediction and planning unit 100, for example, by wireless communication. The sensor 200 is expected to detect various conditions, such as position, acceleration, sound, temperature, vibration, smell, and images.

The output unit 300 is, for example, an interface or display that controls an actuator. An actuator is a mechanical element that constitutes a mechanical/electrical circuit and converts input energy or an electrical signal into physical motion. The actuator is controlled by the calculation results of the prediction and planning unit 100, for example, the action plan results. The display shows the calculation results, for example, the action plan results or the prediction results, so that a person can recognize them.

The prediction and planning unit 100 comprises a parent device, a server 104, and multiple child devices (at least one selected from a DNN 101, a reservoir 102, and an annealing machine 103). The prediction and planning unit 100 performs calculations for prediction and action planning in cooperation with the parent device and child devices through wireless or wired communication, and further through communication between the child devices, and controls actuators based on the calculation results, or displays the results on a display.

In this embodiment, it is assumed that the child devices are composed of independent terminals, so when applying a wired connection between child devices, it is necessary to provide a transceiver (transmitter/receiver circuit) for long-distance transmission, rather than, for example, a bus connection within a semiconductor chip or between chips. That is, a transceiver for long-distance transmission must have a transmitter amplifier (analog circuit) in the transmitter circuit to compensate for attenuation and deterioration of the signal waveform that accompanies long-distance transmission, and a receiver amplifier (analog circuit), an equalizer (waveform equalization circuit), a synchronization circuit, etc. in the receiver circuit.

In FIG. 1, two DNNs 101, two reservoirs 102, and two annealing machines 103 are arranged, but the number of each can be any number, and may be three or more. This is also true for the following embodiments, and the number of child devices can be set arbitrarily as needed. By linking the child devices, it is possible to perform calculations of the desired scale.

The server 104 comprises an input device 1041, an output device 1042, a processing device 1043, and a storage device 1044, as is a known server configuration. In this embodiment, functions such as calculation and control by the server 104 are realized by the processing device 1043 executing a program stored in the storage device 1044, in cooperation with other hardware to perform predetermined processing. The program executed by the server, its functions, or the means for realizing the functions may be referred to as a "function", "means", "part", "unit", "module", etc.

The above configuration may be configured as a single server, or any of the input device 1041, output device 1042, processing device 1043, and storage device 1044 may be configured as other computers connected via a network.

In this embodiment, functions equivalent to those configured by software can also be realized by hardware such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

According to this embodiment, dynamic optimal control of a large-scale system can be realized, and a high sense of realism can be created. The above configuration is a typical example. The server may be a general computer or an MEC server. The sensor 200 may be provided on the server 104 or the terminal side.

<Child device linkage system>
2 is a block diagram showing the overall configuration of an information processing system according to a second embodiment. In this embodiment, the system includes an environmental sensor such as a gas sensor 201 as a sensor, and a camera 202 that captures images of people and equipment. Data from the gas sensor 201 and image data from the camera 202 are collected in a server 104 via a wired or wireless connection. An MEC server may be used as the server here.

The DNN 101, reservoir 102, annealing machine 103, and server 104 each have a wireless interface 105, and are capable of transmitting and receiving data to and from each other. In this embodiment, the hardware configurations of the DNN 101, reservoir 102, and annealing machine 103 are all of the same standard.

Deep learning is a technique for configuring a function approximator to learn a neural network (such as a DNN) consisting of an input layer, a hidden layer, and an output layer, and output an arbitrary vector f(x) for an input vector x. That is, the output is expressed as f1=f1(x1, x2, x3), f2=f2(x1, x2, x3). In this embodiment, the input layer and the output layer are realized by the server 104, and the hidden layer is configured by the DNN 101. Strictly speaking, the DNN 101 corresponds to the hidden layer, which is a part of the deep neural network, but for convenience, the name DNN is used in this specification. The DNN 101 can learn, for example, by supervised learning. In this embodiment, the DNN 101 is assumed to have been trained by a known technique so that it can extract arbitrary features from data from the gas sensor 201 and image data from the camera 202. If the input to DNN101 is time-series data, the output from DNN101 will also be time-series feature data.

Reservoir computing is a technology that is composed of three types of layers, an input layer, an intermediate layer (reservoir layer), and an output layer (readout neuron layer), and that can perform supervised learning, forming a function approximator. In this embodiment, the input layer and the output layer are realized by the server 104, and the reservoir layer is composed of the reservoir 102. The output layer is assumed to have been trained using known techniques. The reservoir 102 performs tasks on time series signals (prediction and feature extraction for time series signals). In other words, the output value at the current discrete time nT is a function of the values of x1, x2, and x3 at the current time nT and past discrete times (n-1)T, (n-2)T, .... In other words, the output of the output layer of reservoir computing is
f1(nT)=f1(x1(nT), x2(nT), x3(nT), x1((n-1)T), x2((n-1)T), x3((n-1)T), x1((n-2)T), x2((n-2)T), x3((n-2)T), ….)
f2(nT)=f2(x1(nT), x2(nT), x3(nT), x1((n-1)T), x2((n-1)T), x3((n-1)T), x1((n-2)T), x2((n-2)T), x3((n-2)T), ….)
...
It is expressed as follows.

Reservoir computing has the advantage that it is easier to learn tasks for time series signals than deep learning. Specifically, the former only needs to learn the weights of the neurons in the output layer. In contrast, the latter (deep learning) needs to learn the weights of neurons in all layers. In the case of DNNs such as RNNs (Recurrent Neural Networks) and LSTMs (Long Short-Term Memory) that process time series signal tasks, learning is performed using a method called Backpropagation through time (BPTT), but as mentioned above, it is necessary to train all layers of neurons corresponding to each time period, which is not easy. Therefore, in one aspect of this embodiment, after the DNN 101 extracts features such as the shape and position of a person and the components of a gas, the reservoir 102 makes predictions based on the changes in these features over time.

The annealing machine 103 is a device that performs optimization calculations, and its basic configuration is disclosed in, for example, Patent Document 2. The annealing machine 103 solves optimization problems by realizing interactions between spins within a spin array consisting of multiple nodes that store binary spins, and transitioning to a minimum or maximum energy state. The spin value can also be expanded to three or more values.

In this embodiment, the following processing patterns are assumed, for example. It is assumed that the processing is performed in real time. Specific application examples include prediction and optimization of people flow within a specified area. In this case, the sensor 200 is a camera or a vibration sensor that acquires image data and vibration data of pedestrians. The DNN 101 detects feature quantities such as the location, facing direction, posture, and speed of each person from the sensor data. The reservoir 102 predicts future data and feature quantities. The annealing machine 103 calculates planned values for, for example, the opening and closing of entrances and the speed of elevators from the current or future feature quantities, and performs optimization such as uniforming the flow of people.

FIG. 3 is a table showing examples of processing patterns possible in the information processing system of FIG.
Pattern 1: Time series data from the sensor 200 is sent to the server 104. The server 104 sends the time series data to the DNN 101, and time series features are extracted. The time series features are sent from the DNN 101 to the reservoir 102 via the server 104. The reservoir 102 predicts future features from the time series features and sends them to the server 104. The server 104 generates an optimization problem based on the future features, converts it into an Ising model that can be processed by the annealing machine 103, and sets the problem in the annealing machine 103. After the problem is solved by the annealing machine 103, the server 104 reads out the solution and outputs it to the output unit 300, for example, as an action plan.

Pattern 2: Time series data from the sensor 200 is sent to the server 104. The server 104 sends the time series data to the DNN 101. The DNN 101 extracts time series features from the time series data and sends them to the server 104. The server 104 generates an optimization problem based on the features, converts it into an Ising model that can be processed by the annealing machine 103, and sets the problem in the annealing machine 103. After the problem is solved by the annealing machine 103, the server 104 reads out the solution and outputs it to the output unit 300, for example, as an action plan.

Pattern 3: Time series data from the sensor 200 is sent to the server 104. The server 104 sends the time series data to the reservoir 102. The reservoir 102 predicts future data or features from the time series data and sends the prediction to the server 104. The server 104 generates an optimization problem based on the future data or features, converts it into an Ising model that can be processed by the annealing machine 103, and sets the problem in the annealing machine 103. After the problem is solved by the annealing machine 103, the server 104 reads out the solution and outputs it to the output unit 300, for example, as an action plan.

Pattern 4: Time series data from the sensor 200 is sent to the server 104. The server 104 sends the time series data to the DNN 101. The DNN 101 extracts time series features from the time series data and sends them to the reservoir 102 via the server 104. The reservoir 102 predicts future features from the time series features and sends them to the server 104. The server 104 outputs the future features to the output unit 300 as predicted values.

Pattern 5: Time series data from the sensor 200 is sent to the server 104. The server 104 sends the time series data to the reservoir 102. The reservoir 102 predicts future data from the time series data and sends it to the server 104. The server 104 outputs the future data to the output unit 300 as a predicted value.

Pattern 6: Time series data from the sensor 200 is sent to the server 104. An optimization problem is generated based on the time series data from the server 104, converted into an Ising model that can be processed by the annealing machine 103, and the problem is set in the annealing machine 103. After the problem is solved by the annealing machine 103, the server 104 reads out the solution and outputs it to the output unit 300, for example, as an action plan.

In this embodiment, the server 104 and child devices 101 to 103 are equipped with a wireless interface 105, and are capable of transmitting and receiving data to and from each other. In particular, by transmitting and receiving the output of the nodes of the child devices between the child devices, the functions of the DNN 101, reservoir 102, and annealing machine 103 can be flexibly configured. Here, the neurons of the DNN 101 and reservoir 102 and the spins of the annealing machine 103 are collectively referred to as nodes.

Compressed data is transmitted between child devices, and the wireless interface 105 of each child device is equipped with a compression/decompression unit. The server 104 ensures synchronization of operations between child devices by broadcasting a synchronization signal to each child device, or by each child device being equipped with an atomic clock. Details of the communication means between the server 104 and the child devices will be described in detail in a later embodiment.

<Example of child device implementation>
4 is a block diagram for explaining an example of circuit implementation of an information processing system. In FIG. 2, the DNN 101, the reservoir 102, and the annealing machine 103 are each shown as an independent child device. However, a plurality of these may be configured on the same child device. That is, for example, any plurality selected from the DNN 101-1, the reservoir 102-1, and the annealing machine 103-1 may be provided as a set on the same child device 410.

To this end, the child device 410 may implement at least two or more functions of the DNN 101-1, reservoir 102-1, and annealing machine 103-1 using a reconfigurable circuit on an LSI (Large Scale Integrated Circuit) or FPGA (Field Programmable Gate Array). This allows one child device to dynamically switch between functions and execute them sequentially, thereby reducing the number of child devices. Alternatively, the DNN 101, reservoir 102, and annealing machine 103 can be implemented in software. In this case, the software is executed by a general-purpose or dedicated processor.

In this embodiment, the gas sensor 201, the camera 202, the output unit 300, the server 104 and the child device are provided with a wireless interface 105 and transmit and receive data wirelessly. A plurality of antennas (or a plurality of wireless transceivers) are connected to the wireless interface 105. In one example, the plurality of antennas can wirelessly communicate in parallel in a plurality of frequency bands. Furthermore, the server 104 is provided with a known spectrum sensing function and determines the communication frequency so as to avoid mutual interference.

<Example of feature extraction on the sensor side>
Fig. 5 is a block diagram for explaining another example of the configuration of feature extraction. In Fig. 2, the DNN 101 is configured as a child device that wirelessly communicates with the server 104, but feature extraction may be performed on the sensor side. In Fig. 5, the DNN 501 is disposed in association with the camera 202 and the odor sensor 203.

In this way, by performing feature extraction using deep learning on the sensor 200 side, the amount of data transmitted from the sensor can be reduced.

<Peer-to-peer connected annealing machine>
6A is a block diagram explaining the concept of a system in which multiple annealing machines 103-1 to 103-4 perform processing in cooperation with each other. In this example, the annealing machines 103-1 to 103-4 connected in a peer-to-peer format and the server 104 are connected to each other wirelessly, similar to FIG. 4A.

The problem setting unit 601, which is a program stored in the storage device 1044 of the server 104, sets an optimization problem based on the features extracted by the DNN 101 or the reservoir 102, or the predicted values from the reservoir 102. An interface that allows user input may be provided for problem setting. The problem setting unit 601 may also set the problem by deep learning. In that case, a deep neural network is provided as hardware.

The problem setting unit 601 converts the optimization problem into an Ising model and determines the external magnetic field coefficient, interaction coefficient, etc. that determine the interaction between spins as known in Patent Document 2 and the like. When the problem is set using deep learning, the feature values extracted by the DNN 101 or the reservoir 102, or the predicted value from the reservoir 102, or the sensor output or a signal processed therefrom, etc. are input to the deep neural network, and the set of coefficients is obtained from the output. The parameters of the deep neural network (such as neuron weights) may be learned and set in advance on the server 104. They may also be learned and set on the server 104 at any time and at an appropriate timing. Alternatively, they may be learned and set in advance or at any time and at an appropriate timing on a computer or cloud outside the server 104. The coefficients determined by the problem setting unit 601 are transmitted from the wireless interface 105 of the server 104 to the annealing machines 103-1 to 103-4. The annealing machines 103-1 to 103-4 set the received coefficients, perform interaction calculations using known methods, and update the spin values.

In a peer-to-peer connection, the annealing machines 103-1 to 103-4 operate as a single annealing machine. To update the value of one spin, the annealing machines use the value of the spin connected to that spin (called an adjacent spin). Therefore, for example, if annealing machine 103-2 has an adjacent spin to update the spin of annealing machine 103-1, the value of that spin will be transmitted wirelessly from annealing machine 103-2 to annealing machine 103-1.

Figure 6B shows a detailed internal block diagram of the annealing machine 103. The wireless interface 105 transmits and receives data. The data compression/decompression unit 604 compresses data to be transmitted and decompresses received data. The spin array 605 is a configuration for transitioning spins to the ground state, as known from Patent Document 2 and the like, and is implemented by applying the principles of semiconductor memory, for example.

Figure 6C is an operation time chart of the annealing machines 103-1 to 103-4 in Figures 6A and 6B. It is assumed that coefficients are set in advance in each of the annealing machines 103-1 to 103-4 from the problem setting unit 601 via the wireless interface 105, and the spin values of the spin array 605 are initialized, for example, randomly.

First, in order to synchronize the annealing machines 103-1 to 103-4, a synchronization signal (S) is broadcast from the synchronization control unit 602 of the server 104 to each annealing machine 103. Each annealing machine 103 is equipped with a synchronization signal receiving circuit and a counter, and resets the counter to zero when it receives the synchronization signal. Thereafter, the counter value is incremented by the internal clock, and each operation is switched based on the counter value. This allows the internal operations of each annealing machine to be synchronized with each other even if the internal clock frequencies of each annealing machine are slightly different.

Next, the order can be arbitrary, but for example, the annealing machine 103-1 performs a transmission process (T) to transmit its own spin value to the annealing machines 103-2 to 103-4, and the annealing machines 103-2 to 103-4 perform a reception process (R). The first transmission (Cycle 1) transmits the initial spin value (usually random). All of the annealing machines' own spin values may be transmitted, but it is more efficient to transmit only the values of the spins that are adjacent to the spins of other annealing machines. The server 104 notifies each annealing machine 103 in advance of the arrangement relationship of the adjacent spins. After that, each annealing machine 103 takes turns in the role of transmitting and receiving. In addition, the transmitted data is compressed by the data compression and decompression unit 604, as described later.

In addition, each annealing machine 103, which is a child device, may attach a timestamp to the transmitted data. This also applies to the other embodiments described below.

After the transmission and reception is completed, the spin array 605 of each annealing machine 103 performs an interaction calculation using a known method, and performs a process (U) to update the spin value. The updated spin value is then transmitted to the other annealing machines 103, but in order to compress the amount of transmitted data, the data compression and expansion unit 604 performs a compression process (P). In the compression process (P), only the difference from the previous time is transmitted.

Then, similar synchronization processing (S), data transmission/reception (T) (R), spin update (U), and data compression (P) are performed from Cycle 2 onwards until the spin values converge or a specified number of times, and each annealing machine 103 transmits the final spin value to the server 104. The solution acquisition unit 603 of the server 104 obtains a solution based on the collected spin values and outputs it to the output unit 300.

In the annealing machine, in order to complete each broadcast in a short time and shorten the calculation time, the data may be divided and transmitted in parallel using multiple frequency channels in each broadcast performed in the time division manner described above. The selection and instruction of the frequency channels is performed by the server 104.

When multiple frequency channels are used, multiple antennas and multiple transceivers are prepared in the wireless interface 105. It is also possible to implement multiple antennas and a common transceiver. In that case, multiple transmission and reception circuits are provided in parallel within the transceiver, and transmission and reception are performed in parallel. To monitor available frequency bands, the server 104 may perform spectrum sensing to grasp available frequency regions on a channel-by-channel basis, and assign them to each child device as channels for the above-mentioned broadcast.

Also, by equipping each child device with multiple antennas and multiple receiving circuits, it is possible for each child device to broadcast to each other simultaneously (at the same time) using frequency division rather than time division. For example, in FIG. 6A, each child device can broadcast simultaneously by using four antennas and four frequency bands, receiving with three antennas and three receivers connected to them, and transmitting with one antenna and a transmitter connected to it. In this case, the communication time can be reduced to one quarter.

Similarly, each child device can be equipped with multiple antennas and multiple transceivers and can broadcast simultaneously by transmitting and receiving using code division multiplexing. In this case, by multiplying a specific spreading code in each transceiver, it is possible to prevent interference while sharing the frequency band. The selection and instruction of the spreading code is performed by the server 104.

In the system of this embodiment, the time it takes to broadcast spin information is longer than the time it takes to update the spin of the annealing machine 103, so it is important to reduce the time it takes to send and receive data between the annealing machines 103. For this reason, it is preferable to speed up transmission using techniques such as frequency division multiplexing, and to compress the spin information before sending it.

One method of compression is to transmit only updated spin information, as mentioned above. To do this, an update map of the spins can be created. In particular, as the calculations progress, the number of updated spins decreases, so the update map is mostly zero (no update) with a small number of 1 (update). Since there are many zeros, compression can be achieved at a high compression rate even with lossless compression methods. Another method is to apply a spatial frequency transform, such as a discrete cosine transform or Fourier transform, to the spin information map to separate it into low-frequency and high-frequency components, and then finely quantize the low-frequency components that are more important with a high number of bits, and roughly quantize the high-frequency components that are less important (or have fewer components) with a low number of bits, thereby increasing the compression rate.

In the compression process (P) of the data compression/expansion unit 604, in addition to or instead of using the update status and difference from the previous state as the transmission data as described above, it is desirable to perform lossy compression of the transmission data. Since the spin values of the spin array are a two-dimensional arrangement of 1 or -1 (0), this can be considered as a monochrome image. Alternatively, the information of multiple spins can be considered as a color image. In the latter case, for example, the information of 24 spins (24-bit information) of 6 x 4 can be considered as one pixel of an 8-bit (R) 8-bit (G) 8-bit (B) color image.

By applying a data compression technique such as JPEG, which is known as an image compression technique, it is possible to losslessly compress the collection data of spins regarded as a monochrome or color two-dimensional image. Because lossy compression such as JPEG has a high compression rate, it is possible to reduce the amount of data transmitted between child devices and shorten the transmission time. The data compression and expansion unit 604 of the annealing machine 103 performs lossy image compression and lossy image expansion. The second annealing machine 103-2 receives the spin information compressed and transmitted by the first annealing machine 103-1 using a lossy image compression method, and performs lossy image expansion to restore the spin information.

Because this is lossy compression, the restored spin information will not be completely the same as the original spin information. However, the annealing machine 103 updates the spins probabilistically and is resistant to random errors, so it is resistant to the above-mentioned spin information errors. Since it is better for the errors to occur randomly, it is possible to have several lossy compression methods and apply them randomly or by switching between them sequentially.

In addition, smoothing filter processing or the like may be applied to monotonize the equivalent image related to the spin information and increase the compression rate. As described above, processing may be performed to extract only the difference data from the previous state. Such filters and processing functions may be provided as part of the data compression and expansion unit 604 or at a stage preceding it. Although some data is lost with smoothing filters and frequency conversion, in the case of an annealing machine, it is believed that this will not have a significant effect on the calculation results.

As described above, various combinations of (with/without generation of update information or difference information) x (with/without compression) are possible for data reduction to shorten transmission times. In addition, since the transmission time is likely to be longer than the time required for data reduction processing, various data reduction methods may be executed serially or in parallel to select the desired result.

The above embodiment is an example of an annealing machine, but it is known that reservoir computers can be broadly divided into two types: ESN (Echo State Network) and LSM (Liquid State Machine). In ESN, the neuron output is a continuous value, so it is expressed in, for example, 32 bits. On the other hand, in LSM, the neuron output is a spike signal, so it is expressed in two or three values such as 0, 1 or -1, 0, 1. In the latter, the output of neurons other than the neuron that generated the spike is 0, so the neuron output information can be compressed at a high compression rate without being converted into update information or difference information, and the transmitted data can be reduced. Also, in the case of ESN, the data reduction effect due to differences is small, so it is similarly better to compress the neuron output information without converting it into difference information.

In the case of a reservoir computer, each neuron is connected not only to nearby neurons but also to distant neurons, so broadcasting is suitable for transmitting neuron output information from the reservoir 102. On the other hand, in the case of an annealing machine, each spin often only needs to be connected to adjacent spins, in which case it is sufficient to transmit output between specific spins.

Figure 6D is a conceptual diagram showing the concept of connecting adjacent spins of an annealing machine. In this conceptual diagram, nine annealing machines 103 are arranged in a two-dimensional array, and each annealing machine has 16 spins 606 (in reality, the number is greater, and there are also cases where the array is three-dimensional). In the case of an annealing machine, it is often sufficient to only couple adjacent spins. In that case, as shown in Figure 6D, the central annealing machine only needs to exchange information on the spins 606 that exist on the boundaries shown in the dotted lines with each of the adjacent annealing machines. For this reason, for example, each annealing machine may use four transceivers in parallel as the wireless interface 105 (or have multiple antennas on one transceiver) and perform simultaneous transmission and reception in four directions by frequency division multiplexing, code division multiplexing, or space division multiplexing. The compression of spin information can be performed using lossy compression, which has a high compression rate, but in the above case, since the amount of transmission is small, lossless compression, which does not degrade information, may also be used.

In the above embodiment, large-scale problems can be handled by combining small-scale annealing machines and transmitting data of adjacent spins between the annealing machines. In addition, large-scale problems can be solved using low-power child devices that are battery-powered or self-generated.

<Star-type connection annealing machine>
7 shows an example different from the fifth embodiment, in which the same problem is solved by a plurality of annealing machines 103-1 to 103-4. The plurality of annealing machines 103-1 to 103-4 perform calculations independently.

The problem setting unit 701 of the server 104 transmits coefficients for the same problem to each annealing machine 103. Each annealing machine 103 performs calculations independently and transmits the results to the server 104. Note that the coefficients, initial values of the spins, and randomness applied during calculations are different between each annealing machine 103.

The solution acquisition unit 703 of the server 104 performs processes such as majority voting and averaging based on the results of each annealing machine to determine the best result. In this embodiment, the annealing machine 103 simply receives data from the server 104 before the optimization calculation and then transmits the results to the server 104, and there is no need for communication between the annealing machines 103. In this embodiment, high-speed parallel solution generation is possible.

<Peer-to-peer connected reservoir>
8A is a block diagram showing an example of a system for wirelessly connecting reservoir computing on a large scale. Reservoirs 102-1 to 102-4 collectively constitute one reservoir layer.
8B shows an internal block diagram of the reservoir 102. The wireless interface 105 transmits and receives data. The data lossless compression and decompression unit 804 compresses data to be transmitted and decompresses received data. The neuron 805 is a collection of neurons that are connected in a known manner to form the reservoir layer.
FIG. 8C shows the functions shared by the server 104 and the reservoir 102 in the system of FIG. 8A, as well as the data to be transmitted and the destination of the data.
FIG. 8D is a time chart showing the operation of the reservoirs 102-1 to 102-4 in FIG. 8A.

The operation will be described with reference to Figures 8A, 8B, 8C, and 8D. The input unit 801 of the server 104 broadcasts input data D to each reservoir 102. If the input data D is an n-dimensional vector, data x1(t) to xn(t) is sent to each reservoir 102. When feature extraction is performed by reservoir computing, the input signal to the reservoir 102 is a sensor signal or a signal to which preprocessing such as a filter has been applied. Data is transmitted and received via the wireless interface 105 (same below).

The synchronization control unit 802 of the server 104 also broadcasts a synchronization signal S. Each reservoir 102 is equipped with a synchronization signal receiving circuit and a counter, and resets the counter to zero when it receives the synchronization signal. Thereafter, the counter value is incremented by an internal clock, and each operation is switched based on the counter value, thereby ensuring synchronization between the reservoirs 102.

Each reservoir 102 broadcasts (T) the output value of the neuron contained in its own reservoir at the current time to the other reservoirs and the server 104. At this time, the order is arbitrary, but for example, reservoir 102-1 performs a transmission process (T) to send the output of its own neuron to reservoirs 102-2 to 102-4, and reservoirs 102-2 to 102-4 and server 104 receive this (R). After that, each reservoir 102 takes turns in the role of sending and receiving.

Each reservoir 102 updates (NU) the neuron value based on the neuron values from other reservoirs and the input signal from the server 104. Each reservoir 102 compresses (P) the updated neuron output information and moves on to the next cycle of processing. Figure 8D shows a time chart for two cycles, but this processing is basically executed repeatedly at the timing of data input to the input section (input layer) of the server 104.

In the time chart of Figure 8D, the part from the synchronization signal + input data value distribution (S/D) to compression (P) is the processing for the input value (one cycle of data) at one time. Therefore, the value of each neuron (node) is updated for one cycle based on the above input data and the current output value of itself and other neurons. For this reason, each reservoir 102 needs to collect all (or part of) the neuron values at the current time. In this system, the neuron values of other reservoirs 102 can be collected by wireless transmission. To complete the broadcast in a short time, frequency division multiplexing or code division multiplexing may be used to broadcast simultaneously, as explained in the fifth embodiment.

The output of the neuron of each child device can be considered as a two-dimensional image, so in the compression process (P), the data lossless compression and expansion unit 804 compresses (P) the data using lossless compression such as GIF (Graphics Interchange Format) and transmits it. On the receiving side, the received data is decompressed to obtain the current output value of the neuron 805 on the transmitting side. This makes it possible to update the neuron 805. In the case of the reservoir, since the calculation method is relatively vulnerable to neuron output errors, a lossless compression method such as run-length coding, which does not degrade information, is preferable to lossy compression.

The neuron output of the reservoir 102 is, for example, a 32-bit continuous value. In order to reduce the overhead of transmitting and receiving neuron outputs between the reservoirs 102, it is effective to broadcast the important upper bits preferentially. Therefore, the transmission method is as follows:
The most significant bit of neuron 1, the most significant bit of neuron 2, ..., the most significant bit of neuron N,
The second bit of neuron 1, the second bit of neuron 2, ..., the second bit of neuron N,
The third bit of neuron 1, the third bit of neuron 2, ..., the third bit of neuron N,
...
The most significant bit of each neuron is transmitted preferentially, as shown below. This transmission method can be adopted instead of data compression.

As mentioned above, when each child device receives a synchronization signal from the server 104, it resets its own counter to zero, and thereafter, the counter is incremented based on the child device's internal clock. When the counter reaches a predetermined value, it is considered to have timed out and the transmission ends. With this method, it is not guaranteed that all information can be broadcast, but by limiting the communication time, the overall processing time is shortened, and by transmitting from the most significant bits as described above, the deterioration of accuracy is also limited. The upper limit of the counter for determining that a time has expired can be dynamically adjusted by the server 104 when assessing the outcome of the task, so that if it determines that higher speed is required, it is made smaller (shorter), or if it determines that higher accuracy is required, it is made larger (longer).

The output unit 803 of the server 104 receives the output of each reservoir 102 and functions as an output layer. In reservoir computing, the output layer receives the output of all or some of the neurons in the reservoir layer and performs product-sum operations using learned weights. In this embodiment, the server 104 plays the role of the output layer, so each reservoir 102 transmits the output of the neurons contained therein to the server 104, which is the output layer. The output unit 803 of the server 104 outputs a time-series output signal every cycle.

In this embodiment, no feedback is provided from the output layer of the server 104 to each reservoir 102, but the output from the output section 803 of the server 104 may be wirelessly transmitted to at least one of the reservoirs 102.

<Star-type connection reservoir>
9A is an example of a star-shaped connection in which the same processing is performed by multiple reservoirs 102-1 to 102-4, which is different from the seventh embodiment. The multiple reservoirs 102-1 to 102-4 share the roles of the reservoir layer and the output layer, and perform calculations independently. Note that each reservoir 102 is different from the others in the connection pattern between neurons in the reservoir layer, the parameters (weights, etc.) and activation function shape of the neurons in the reservoir layer, the initial value of the output of the neurons in the reservoir layer, and the parameters (weights, etc.) of the neurons in the output layer.
Fig. 9B shows the functions shared by the server 104 and the reservoir 102, the data to be transmitted, and the destination of the data in the system of Fig. 9A. In this example, the reservoir 102 is responsible for the output layer, but if the reservoir 102 does not have an output layer, the server 104 may be provided with an output layer for each of the four child devices.
FIG. 9C is a time chart showing the operation of the reservoirs 102-1 to 102-4 in FIG. 9A.

The operation will be described with reference to Figures 9A, 9B, and 9C. The input unit 901 of the server 104 broadcasts the same input data to each reservoir 102 (D). Each reservoir 102 performs calculations independently (NU) and transmits the results from the output layer to the server 104 (T). During transmission, interference is prevented using time division multiplexing, code division multiplexing, frequency division multiplexing, space division multiplexing, etc.

The solution acquisition unit 903 of the server 104 determines the best result by taking a majority vote or averaging the results of each reservoir 102. High-speed parallel solution generation is possible.

In this embodiment, the same problem can be solved in multiple reservoirs and the best result determined on the server, enabling high-speed parallel calculations.

<Hierarchical Connection DNN>
10A is a block diagram showing an example of a system for expanding the scale of a DNN by wireless connection. DNNs 101-1 to 101-4 together constitute a single DNN.
FIG. 10B shows the functions shared by the server 104 and the DNN 101 in the system of FIG. 10A, as well as the data to be transmitted and the destinations thereof.

The operation will be explained with reference to Figures 10A and 10B. The input unit 1001 of the server 104 transmits input data to the DNN 101-1. If the input data D is an n-dimensional vector, data x1(t) to xn(t) is transmitted.

In deep learning, DNNs 101-1 to 101-4 are connected in series, and data is transmitted sequentially from the previous layer. In the figure, the hidden layer is composed of four layers, but the number of layers can be changed depending on the task. For simple tasks, a small number of layers is sufficient, but the number of layers increases as the task becomes more complex. Unlike reservoir computing and annealing machines, deep learning can perform complex tasks by increasing the number of layers. Therefore, the desired task can be achieved by connecting child devices in series to ensure the required number of layers. Also, in the figure, each child device has one convolutional layer and one pooling layer, but it may have more layers or various types of layers.

Wireless communication between DNNs 101-1 to 101-4 may be performed using frequency division multiplexing, code division multiplexing, or space division multiplexing to prevent interference. For high-speed communication, multiple antennas, transceivers, and frequency channels may be used to transmit and receive in parallel.

The synchronization control unit 1002 of the server 104 also broadcasts a synchronization signal S. Each DNN 101 is equipped with a synchronization signal receiving circuit and a counter, and resets the counter to zero when the signal is received to ensure synchronization between the DNNs 101.

<Star-type connection DNN>
In another example different from the ninth embodiment, a star-shaped connection is possible in which multiple DNNs 101 perform the same feature extraction task. Multiple DNNs 101-1 to 101-4 share the role of hidden layer and output layer, and perform calculations independently. Server 104 shares the role of input layer. Each DNN 101 differs from the others in terms of pruning points of neurons and connections between neurons, neuron parameters (weights, etc.), shapes of activation functions, initial values of neuron outputs (in the case of recurrent deep neural networks), etc.

The input layer of server 104 broadcasts input data to DNNs 101-1 to 101-4. DNNs 101-1 to 101-4 perform calculations independently and in parallel, and transmit the results to the solution acquisition unit of server 104. During transmission, interference is prevented using time division multiplexing, code division multiplexing, frequency division multiplexing, space division multiplexing, etc.

The solution acquisition unit of the server 104 determines the best result by taking a majority vote or averaging the results of each DNN 101. This enables high-speed parallel solution generation.

In this embodiment, the same problem can be solved using multiple DNNs and the best result can be determined on the server, enabling high-speed parallel calculations.

<Ensemble-type DNN>
In deep learning, it is known that multiple different DNNs (called ensembles) are used to perform separate calculations for one task, and the results of these calculations are combined to improve inference accuracy. Example 10 is also based on the ensemble technique, but in this example, multiple DNNs that have learned parameters (such as neuron weights) using different data sets are used as the ensemble. These DNNs have different network structures and different weight parameter sets.

Multiple DNNs are used as an ensemble in a star-shaped connection similar to that of Example 10, and the server 104 broadcasts input signals corresponding to a single task to each DNN, and also receives and combines the output results of each DNN to improve inference accuracy.

In the information processing systems described in the first to eleventh embodiments, synchronization of each child device is performed based on a synchronization signal sent from the server. As an alternative method, each device may be provided with a clock that can accurately measure absolute time, such as an atomic clock, and may operate based on the time of the atomic clock or the like. Also, to guarantee the chronological relationship of the data being sent and received, each device may include a timestamp in the data being sent.

Each of the above DNNs may compress and expand data transmission and reception in the same way as the reservoir.

<Simultaneous multi-connection pipeline>
In the information processing system described in the first to eleventh embodiments, in order to connect the entire wide-area social system and control the entire system without delay, it is necessary to collect data, process the data, and control the system in real time. In the wide-area social system, individual devices are physically distributed and arranged, and the communication means and distances between them are not necessarily uniform. Therefore, it is desirable to connect data transmitted and received through multiple communications with multiple edge AIs (DNN 101, reservoir 102, annealing machine 103, etc.) while ensuring synchronicity.

Figure 11 is a conceptual diagram showing an example of simultaneous real-time processing of multiple data by the prediction planning unit 100. Multiple data 1 to 5 transmitted from the sensor 200, etc., may be connected via a mixture of wireless communications of different specifications, such as 5G and 6G, and wired communications of different specifications. The physical locations of the senders, such as the distance from the senders, may also differ. Furthermore, the connection relationships and the conditions of upstream communication, downstream communication, side link, etc. may not be the same. In this way, the received data includes at least one of data obtained from different sensors, data that has passed through different routes, and data by different communication means, so the latency of each data is different, and it is considered that the order in which the data arrives at the prediction planning unit 100 will not be the order in which the data was generated.

In this case, if the data is stored once as a database and then processed, the order of the data can be guaranteed, but real-time processing will not be possible. In this embodiment, the function of the simultaneous multi-connection pipeline 1100 is added to enable real-time data processing. The simultaneous multi-connection pipeline 1100 has the functions of a data flow controller 1110 and a time-space organizer 1120. The data flow controller 1110 manages the order of data analysis and transmits the data to the forecast planning unit 100 in the order in which it should be processed. The setting of the data flow controller 1110 is controlled, for example, by the AI orchestrator 1045. The time-space organizer 1120 labels the data generation time and data generation position of the received data (for example, various sensor data) based on, for example, the measurement results of the arrival time of the received data and the measurement results of the time required for arrival, the timestamp of the received data, the measurement results of the sensor position, etc.

<Hierarchical Edge>
FIG. 12 is a block diagram showing an example of an edge 1200 configured with a combination of the prediction planning unit 100 and the simultaneous multiple connection pipeline 1100 in FIG. 11, and having a hierarchical structure. The prediction planning unit 100 and the simultaneous multiple connection pipeline 1100 may be physically separated. Each of the edges 1200a-f may be configured based on the same reference architecture. The edges 1200a-f can define the necessary functions by software according to their roles. The edges 1200a-f are configured, for example, by a general server or computer, or dedicated hardware, and the functions of the edge 1200 are implemented by software.

In FIG. 12, an example is disclosed in which each edge 1200 includes a prediction planning unit 100. The prediction planning unit 100 includes a parent device (server 104) and child devices 101 to 103, as disclosed in FIG. 1, for example. However, the parent device and the child devices do not necessarily need to exist in the same edge. For example, the child devices 101 to 103 of edge 1200a may be controlled by a server 104 of an MEC server or cloud edges 1200d to f. In this case, the server 104 does not necessarily need to be placed in edge 1200a. As described above, the parent device that controls the child devices can be placed in any edge among the hierarchical edges, taking into account the required processing volume and communication volume.

The edges 1200a-c are, for example, sensor edges that are placed near a site 1210 that includes a person 1211, a robot 1212, various devices 1213, transportation facilities 1214, etc. that are to be controlled. The edges 1200a-c are equipped with a sensor unit 1220 and, if necessary, a pre-processing unit 1230. The pre-processing unit 1230 performs desired processing such as filtering, data compression, data division, and encoding. As described in FIG. 5, feature extraction may be included in the pre-processing. By performing pre-processing on the sensor side, the amount of data to be transmitted can be reduced.

The sensor unit 1220 may include various sensors and cameras, as well as the functions of a PLC (Programmable Logic Controller) and XR (X Reality). The PLC and XR function as a means of collecting data, just like the sensors and cameras. The various sensors, cameras, PLC, and XR of the sensor unit 1220 not only collect and output data from robots and the like, but can also be control targets to which input from the edge 1200 is fed back. For example, the frequency of data acquisition and data transmission is changed by feedback. In this way, it is possible to control the sensor unit 1220 accordingly in accordance with the information collection and control of robots and people.

In one example, data from the sensor unit 1220 is sent to the edges 1200a-f either directly or after being processed by the preprocessing unit 1230. The data is processed by the edge 1200, and the results of inference by the prediction and planning unit 100 are displayed by the output unit 300, or fed back to the robot 1212, various devices 1213, transportation facilities 1214, or the sensor unit 1220 at the site 1210. In another example, the output of one or more edges 1200 may be used as the input of other edges.

It is assumed that various wired and wireless communication methods will be used in the network NW. Various wireless communication methods are possible, such as 4G, P-LTE, 5G, Beyond 5G, 6G, and Wi-Fi (trademark). Therefore, as explained in FIG. 11, the latency of transmitted data varies depending on the positions of the edges 1200a-c and the communication method used.

The edges 1200d-e are, for example, MEC servers that are placed closer to the site 1210 than the cloud. Being closer to the site, they can reduce delays compared to the cloud. For example, the edges 1200d-e are installed on base stations in the network NW.

The edge 1200f is configured, for example, as a cloud server that is installed in the cloud or at the entrance to the cloud. The edge 1200f may be accompanied by an overall control unit 1240 or connected via a network. The functions of each edge 1200a to f may be deployed, for example, via the overall control unit 1240. The deployment procedure may follow conventional technology.

The prediction and planning unit 100 and simultaneous multi-connection pipeline 1100 of each edge 1200a-f included in this system are deployed with the necessary functions for each edge. For example, the prediction and planning units 100a-c of edges 1200a-c close to the site 1210 perform simple processing (e.g., recognition), while the prediction and planning unit 100f of edge 1200f in the cloud performs complex processing (e.g., understanding and judgment). The simultaneous multi-connection pipeline 1100 forms a data flow that is compatible with the processing performed by the prediction and planning unit 100.

The simultaneous multi-connection pipeline 1100 guarantees the time order of data required for processing by the prediction and planning unit 100. It is desirable to perform the time order of data strictly (e.g., in milliseconds) for, for example, controlling the operation of industrial robots. Furthermore, such strictness is not required for macro-level control of people flow (e.g., in minutes). In this embodiment, the specifications of the prediction and planning unit 100 and the simultaneous multi-connection pipeline 1100 can be optimized for each edge.

<Edge details>
Fig. 13 is a block diagram showing details of an edge 1200. Although only one edge 1200 is shown in Fig. 13, the overall control unit 1240 may manage a plurality of edges 1200 as in Fig. 12. The edge 1200 includes a simultaneous multi-connection pipeline 1100 and a prediction planning unit 100. The data flow is indicated by a thick arrow, and the control flow is indicated by a unidirectional or bidirectional thin arrow.

The prediction and planning unit 100, as in the embodiments already described, comprises any combination selected from the DNN 101, the reservoir 102, and the annealing machine 103. The prediction and planning unit 100 also comprises an AI orchestrator 1045 and a feedback and feedforward unit 1046. The AI orchestrator 1045 and the feedback and feedforward unit 1046 are part of the functions of the server 104, and are stored as programs in the storage device 1044.

The simultaneous multi-connection pipeline 1100 includes the data flow controller 1110 and the time-space organizer 1120 described in FIG. 11. The simultaneous multi-connection pipeline 1100 also includes a synchronous communication measurement unit 1101, a position and speed measurement unit 1102, and a control flow controller 1103. The function of the simultaneous multi-connection pipeline 1100 is part of the function of the server 104, and is stored as a program in the storage device 1044. In this case, the AI orchestrator 1045, the feedback and feedforward unit 1046, and the simultaneous multi-connection pipeline 1100, which are shown as separate blocks in FIG. 13, are implemented in software on the same server 104. Alternatively, the simultaneous multi-connection pipeline 1100 may be configured on a server other than the server 104, and connected to the forecast planning unit 100.

The overall control unit 1240 includes an operation management DB 1241, a composite AI 1242, and an asset control unit 1243 that includes a GUI (Graphical User Interface).

For example, when a user who is an administrator of the site 1210 implements a desired function in the edge 1200, the user can configure the prediction planning unit 100 and the simultaneous multi-connection pipeline 1100 via the overall control unit 1240. The operation management DB 1241 stores the configurations of the subordinate prediction planning units 100a-f and simultaneous multi-connection pipelines 1100a-f. The configurations of the prediction planning unit 100 and the simultaneous multi-connection pipelines 1100 are transmitted to the AI orchestrator 1045.

The AI orchestrator 1045 selects one of the patterns 1 to 6, selects the operation type of the DNN 101, the reservoir 102, the annealing machine 103, etc. based on the pattern, specifies the connection type (peer-to-peer connection or star connection) for each operation type, selects the child device to be used, and specifies whether or not a connection is required for each connection between the child devices. The AI orchestrator 1045 also instructs the pipeline operation management unit 1300 on the order of data to be input to the forecast planning unit 100. The order of data can be the order of arrival, the order of transmission time, the order of distance, etc. These settings are made by the asset control unit 1243 or the user terminal 1305 connected by a network according to the processing contents of the DNN 101, the reservoir 102, the annealing machine 103, etc. The operation management DB 1241 also has a backup function as a mirror site for recovery in the event of a problem with the edge 1200. Learning of the DNN 101 and the reservoir 102 may be performed by the overall control unit 1240 or by the edge 1200.

The composite AI 1242 performs calculations that do not need to be performed on the edge 1200 or that are better performed in the cloud. An example of a calculation that does not need to be performed on the edge 1200 is a case where a delay of a certain amount or more is acceptable. For example, operations that do not deviate from routine operations can be processed in the cloud. An example of a calculation that is better performed in the cloud is predictions using big data in quantities that cannot be stored on the edge. Large amounts of data will be stored on the cloud and processed by the composite AI 1242. The acquired data and calculation results will also be stored in the cloud.

The edge 1200, which is equipped with the DNN 101, reservoir 102, annealing machine 103, etc., can process data from the sensor unit 1220 or other edges 1200.

The pipeline operation management unit 1300 includes an edge management DB 1301, a communication control unit 1302, a transmission time DB 1303, and a time-space DB 1304.

The edge management DB 1301 manages basic information on people 1211, robots 1212, various devices 1213, transportation facilities 1214, etc., that are deployed at the site 1210 connected to the edge. It also manages basic information on the sensor unit 1220 that is similarly connected. It also manages basic information on the communication line used for the connection. The basic information is the standards and specifications of each device and line.

The edge management DB 1301 also manages the correspondence between the data flow sent from the sensor unit 1220 and the input terminals of the DNN 101, the reservoir 102, the annealing machine 103, and the server 104. The correspondence is set based on information about the configuration of the sensor unit 1220 and information about the configuration of the DNN 101, the reservoir 102, and the annealing machine 103. It can also be set by the user. For example, in the examples of Examples 10 and 11, the same data flow may be transmitted (broadcast) in parallel to each DNN 101, or the data flow may be transmitted first only to the server 104, and then transmitted in parallel from the server 104 to each DNN 101. In the example of Example 7, the same data flow may be transmitted in parallel to each reservoir 102, or first to the server 104, and then transmitted in parallel from the server 104 to each reservoir 102. In the example of Example 9, the data flow may be transmitted only to the DNN 101-1, or only to the server 104. Information in the edge management DB 1301, such as information for controlling data flow, is input from the operation management DB 1241 via the AI orchestrator 1045 or separately. In this way, how the data flow controller 1110 rearranges, divides, and transmits the data flow depends on the configuration of the sensor unit 1220 and the configuration of the child device of the forecast planning unit 100.

The communication control unit 1302 controls the timing of measurement and notification of data sent from the sensor unit 1220 using known technology via the control flow controller 1103. It also supervises the entire communication processing performed by the edge 1200. The settings of the communication control unit 1302 are input from the operation management DB 1241 via the AI orchestrator 1045 or separately.

The transmission time DB 1303 and the time-space DB 1304 store data collected in association with the operation of the edge 1200, as described below. For backup purposes, etc., the data in the pipeline operation management unit 1300 is appropriately copied to the operation management DB 1241.

The synchronous communication measurement unit 1101 is equipped with a reference clock such as a quartz clock or an atomic clock. The synchronous communication measurement unit 1101 broadcasts the time of the reference clock to the sensor unit 1220 that is the data transmission source. The channel for broadcasting is, for example, a 5G wireless channel. Although it will increase costs, each sensor unit 1220 may be equipped with an atomic clock or the like, and broadcasting may be omitted. With this configuration, each sensor unit 1220 can be equipped with a common clock, so that the transmitted data can be time-stamped based on a common time axis. The synchronous communication measurement unit 1101 can calculate the transmission delay time (time required for transmission) of data from each sensor unit 1220 from the difference between the timestamp of the received data and the time of its own reference clock at the time of receiving the data. The transmission delay time of data corresponding to each sensor unit 1220 is recorded in the pipeline operation management unit 1300 as a transmission time DB 1303.

The position and speed measurement unit 1102 measures the position and speed of the person 1211, robot 1212, various devices 1213, transportation 1214, or the sensor of the sensor unit that is the control target. The control target can measure its own position and moving speed using its Global Positioning System (GPS) or Global Navigation Satellite System (GNSS). Since the position accuracy of GPS and GNSS is on the order of one meter, RTK (Real Time Kinematic) or the like is used to obtain even higher accuracy. In RTK, the position information of GPS or GNSS is acquired at two locations, the control target and the position and speed measurement unit 1102, and the position information of the position and speed measurement unit 1102 is broadcast to the control target side, and the difference is used to correct the deviation of the position information, thereby enabling an accuracy of several centimeters. In addition, the position information can be corrected using information from a camera that is photographing the control target or millimeter waves emitted from base stations used in 5G and post 5G. With this configuration, accurate location information of the sender can be added to the data transmitted from each sensor unit 1220. Furthermore, the transmitted data can also include data acquired by various sensors such as weight sensors, optical sensors, image data, and thermal sensors instead of or in addition to location information.

The time space organizer 1120 labels the received data with time information, location information, and other sensor information. Specifically, the data transmission time (which can be considered to be equal to the data generation time) is calculated by subtracting the delay time in the delay time data table from the arrival time of the received data. The calculated data generation time and location information are added to the received data as a header. This makes it possible to convert the received data collection time and location into four-dimensional coordinates, and a time space DB 1304 can be generated in the pipeline operation management unit 1300. The time space organizer 1120 can also monitor delays and jitters from the transmission delay time of data measured by the synchronous communication measurement unit 1101. The sensor units 1220 can also be classified and sliced based on delays and jitters.

The data flow controller 1110 sorts the received data so that analysis is performed in the order of data generation time for real-time processing in the prediction planning unit 100. For this reason, the data flow controller 1110 has a buffer function for sorting the received data based on the labeling given to the data by the time-space organizer 1120. As a specific implementation example, the data is sorted by temporarily storing the data in a work memory such as a DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory) as a data buffer. The data flow controller 1110 may also add a destination header to the sorted data according to the information in the edge management DB 1301 and transmit it to multiple DNNs 101, reservoirs 102, and annealing machines 103.

In addition to the above-mentioned mode of sorting data in chronological order, the data flow controller 1110 may also send data to the prediction and planning unit 100 in the order of arrival. This mode is applicable when the chronological order of data is not strictly required, and prediction and action plan results can be obtained in the shortest time. Alternatively, data from a specific sensor may be sent preferentially to the prediction and planning unit 100 in order to perform analysis focusing on the specific position of the sender. For example, when monitoring robots or drones on an entire floor of a factory, etc., if a change occurs in one particular robot, it may be desirable to identify and trace the position of that robot and analyze its behavior as a priority.

The control flow controller 1103 distributes control signals to the controlled objects of the site 1210 and the sensors of the sensor unit 1220 in response to instructions from the communication control unit 1302. The control signals are feedback signals generated by the feedback and feedforward unit 1046 based on the outputs of the DNN 101, the reservoir 102, and the annealing machine 103, or feedforward signals generated by the feedback and feedforward unit 1046 in response to instructions from the asset control unit 1243.

As described above, the asset control unit 1243 serves as a user interface for setting the contents of the edge 1200, and also serves as an interface for the user to directly control the control target of the site 1210 and the sensor of the sensor unit 1220 based on the prediction results output by the edge 1200. The asset control unit 1243 may be configured to perform the desired control based on input from the user terminal 1305.

<Details of AI learning>
As is well known, the DNN 101 and the reservoir 102 constitute a network of neurons, and after learning parameters such as weights, inference, i.e., execution of AI, becomes possible. Several learning methods that can be adopted in this embodiment are described below.

In the first method, a configuration (replica) identical to the neuron network configuration of the DNN 101 and reservoir 102 is formed within the server 104 in a closed form within the server 104, and learning is performed. Since learning is performed only within the server 104, learning can be performed quickly. The replica can be generated by the AI orchestrator 1045 by imitating the actual structure of the DNN 101 and reservoir 102, including the connections between child devices.

In the case of the reservoir 102, a replica of the reservoir layer network is prepared in the server 104, and learning is performed using the original input layer and output layer in the server 104 (see FIG. 8C). As training data, input training signals and output training signals corresponding to the task are generated in the server 104. Information required for generating training data is, for example, past data stored in the operation management DB 1241 that is read and used. Only the parameters of the output layer (weights, etc.) are learned. The parameters of the neurons in the reservoir layer (weights, etc.) and the corresponding parameters in the replica configuration are fixed to values that are randomly set in advance. Since only the parameters of the output layer in the server 104 are learned, there is no need to send the learning results to each reservoir 102.

In the case of DNN101, as in the case of reservoir computing, learning can be performed using a DNN replica configuration separately provided in server 104. In learning a DNN, the parameters (weights, etc.) of all or some of the neurons are learned. Unlike the case of reservoir computing, the learned parameters must be reflected in the corresponding neurons of the corresponding DNN101, so information on the learned parameters is transmitted from server 104 to each DNN101 using, for example, wireless interface 105, and the parameters are set.

In the case of reservoir computing, it is possible to learn the parameters of the neurons in the reservoir layer, just as in the case of DNN. In that case, just as in the case of DNN, the learned neuron parameters can be transmitted from the server 104 to each reservoir 102 and set.

The second method uses the neuron networks of DNN 101 and reservoir 102 as is, and learns with the same configuration as during inference.

In the case of reservoir computing, an input teacher signal and an output teacher signal are generated within the server 104, and the input teacher signal is broadcast to each reservoir 102 via the wireless interface 105. The server receives the neuron output of each reservoir 102 and performs processing in its own output layer. The output signal of the output layer (i.e., the output signal as reservoir computing) is compared with the output teacher signal, and the parameters of the output layer (weights, etc.) in the server 104 are updated so that the difference approaches zero. Since parameter updates occur only within the server 104, there is no need to transmit data from the server 104 to each reservoir 102 when updating, and learning can be performed quickly.

In the case of DNN (assuming child devices are connected in series), an input teacher signal and an output teacher signal are generated within the server 104, and the input teacher signal is sent to DNN 101-1, which serves as the first stage of the DNN (see Figures 10A and 10B). The server 104 receives the output of DNN 101-4, which serves as the final stage of the DNN (i.e., the output as deep learning). This output signal is compared with the output teacher signal, and the parameters of each DNN 101 (such as neuron weights) are updated so that the difference approaches zero. To update the parameters, transmission from the server 104 to each DNN 101 is necessary. Therefore, learning is slower than learning by replicas.

The first method, which uses a replica in the server, is inferior in terms of accuracy to the second method, which uses an actual device, because deviations in characteristics occur when the circuit configurations of the child device and the parent device (server) are not completely identical. In particular, when the child device is implemented with an analog circuit that aims for low power consumption, deviations occur between the replica in the server and the characteristics (weight values, etc.) of the actual child device due to manufacturing variations in the analog circuit and fluctuations in circuit characteristics due to fluctuations in temperature and power supply voltage. With the second method, regular learning is performed using the actual child device, making it possible to track characteristic fluctuations and maintain high inference accuracy. On the other hand, the method using a replica has the advantage of enabling high-speed learning, as mentioned above.

The above learning can be performed before the DNN 101 and reservoir 102 are put into actual operation, with tasks anticipated in advance, or can be performed at any time during actual operation to accommodate new tasks. Additionally, learning can be repeated periodically using the above method as additional learning. Additional learning can accommodate changes in on-site conditions, changes in sensor characteristics, or changes to the settings of the data flow controller, which will be described later.

Note that, although an example of supervised learning has been described above, reinforcement learning or other known learning methods may also be adopted. In reinforcement learning, while actually performing inference, the server 104 evaluates the current AI results using some kind of index, and updates the weights of the reservoir output layer and the weights of the DNN based on the evaluation results. The learning functions described above may be included as part of the functions of the AI orchestrator 1045.

<Step-by-step compound AI>
FIG. 14 is a block diagram showing the concept of staged composite AI using the system configurations of FIG. 12 and FIG. 13. The data flow controller 1110 described in the previous embodiment rearranges the data flow in the order of actual event occurrence, so that data that accurately reflects the real time-space can be generated, enabling accurate inference and prediction. On the other hand, since the data flow controller 1110 is a kind of buffer, data delay occurs in principle. Therefore, there is a trade-off between accuracy and delay time. Also, in the early stages of operation, there are cases where priority is given to obtaining some kind of inference result. Therefore, in this embodiment, an example is shown in which the data flow controller 1110 can control the data flow depending on the situation and stage. As a specific example, when rearranging data based on the transmission timing of the sensor side, the proportion of data to be rearranged is changed, and the accuracy and speed of processing due to the change in the proportion of data to be rearranged can be evaluated.

First, the data flow controller 1110 cancels the data sorting control and has the forecast planning unit 100 process the data in the order of arrival (S1401). In this case, in principle, it is possible to perform calculations at the highest speed with low latency (S1402).

Next, the data flow controller 1110, for example, sorts only the data flows from one or more specified sensors in the order of occurrence according to the settings of the AI orchestrator 1045 (S1403). The settings for which sensors' data flows are to be sorted may be determined in advance, or may be changed dynamically by the user. As a specific example, when changing the proportion of data to be sorted, if there are multiple types of sensors on the sensor side, the data is sorted for each sensor type and controlled separately from data that is not to be sorted. In addition to controlling for each sensor type, control may be performed for each sender, each line, or based on any grouping. These can be identified based on the sender address or flags in the data flow.

As a result, depending on the control conditions, delays may increase slightly, but accuracy may improve. The delay time and accuracy are evaluated, for example, by the overall control unit 1240, by comparing them with predetermined ideal delay time and accuracy conditions (S1404). As described above, the data flow controller 1110 tries several (or all) combinations of controlled and uncontrolled data flows.

In one method, all combinations are tried and the one with the best evaluation is set as the setting for the data flow controller 1110. In the above, an example was shown in which data flow control was switched on and off for each sensor, but other combinations are also possible, such as for each device to be controlled or for each site location. For example, priorities can be assigned in advance to parts (or sensors) for which accuracy is important, and the data flow can be rearranged chronologically in order of priority.

In the above example, the order of the data flows input to the actual child device is changed, but if the second method of learning using a replica described in Example 14 is adopted, the changed data flows can be input to the replica and used for evaluation, and the actual child device can be switched to the best conditions obtained from the replica and the data flow controller 1110 can be operated. With this method, the output of the prediction planning unit 100 will gradually approach the target conditions for delay time and accuracy.

Another method is to gradually increase the proportion of data flow to be controlled (e.g., monotonically increase), and fix the settings of the data flow controller 1110 when the delay time reaches a preset upper limit. In this case, the accuracy will be improved step by step within the allowable range of delay time.

<Future prediction feedforward AI>
Fig. 15 is a block diagram showing the concept of parallel execution of current analysis and future prediction by the system configurations of Fig. 12 and Fig. 13. In the hierarchical edge configuration as shown in Fig. 12, the capabilities and specifications of the edges of each layer, as well as the amount and transfer speed of data provided to the edges, are different, so that it is possible to share processing suitable for the edges of each layer.

The composite AI 1242 (consisting of a DNN, a reservoir, an annealing machine, etc.) of the overall control unit 1240, which is close to the cloud, has the resources to use and process big data, but is located away from the site, which puts it at a disadvantage in obtaining real-time data. Therefore, the composite AI 1242 uses past history data from the DB, rather than real-time sensor data, to make rough future predictions of routine operations that do not require consideration of sudden events (S1501).

The prediction planning unit 100 uses the DNN 101 and reservoir 102, or the reservoir 102, to make predictions for the future of about 100 msec (patterns 4 and 5 in Figure 3). In addition, the DNN 101 or reservoir 102 and the annealing machine 103 are used to plan actions and trajectories (patterns 1 to 3, 6, S1502 in Figure 3). Real-time sensor data is used as input for this process.

The annealing machine uses the time series output for multiple (e.g. 10,000) time steps obtained from DNN feature extraction, reservoir feature extraction, and prediction to set up and solve a single optimization problem. In this case, the solution throughput is 10,000 times slower, but it is possible to select the optimal action for a more complex state.

For example, DNN 101 and reservoir 102 extract features of people flow in a city for five minutes, and then the server 104 uses the results of those five minutes to set an optimization problem for equalizing people flow over the next five minutes (i.e., determines the coupling coefficients between the spins of the annealing machine 103). Furthermore, the above coupling coefficients are sent to the annealing machine (problem setting complete), and the annealing machine solves the above problem over the next five minutes and returns the results to server 104. Based on the results, server 104 displays recommended routes on a display placed on a street corner, for example. It is conceivable that congestion in the city can be alleviated by periodically repeating the above 15-minute operation.

However, as is clear from the above explanation, processing using an annealing machine takes longer than processing using only DNN 101 or reservoir 102. Feature extraction and prediction using only DNN 101 or reservoir 102 allows for more real-time processing and response.

For example, it would be possible to estimate what a person will want to do next by extracting characteristics of their movements or by predicting it, and provide support (or instructions) in advance using actuators, or support using a wearable display (such as showing the location of necessary tools). For example, when a person is attempting to perform non-routine tasks remotely, it would be good to be able to predict movements within 200 msec, which is the sensitivity of the brain, in order to prevent any discomfort during the time it takes for the support to be transmitted. Predictions could be made to detect not only signs of abnormalities, but also signs of non-routine tasks. Furthermore, future predictions could be used to conceal communication delays that occur when communicating to the other side of the world or with artificial satellites.

To further speed up the response, only feature extraction of the DNN 101 and reservoir 102 is performed using real-time sensor data at the edge close to the site, enabling equipment abnormalities to be quickly recognized and emergency shutdowns to be implemented in the event of a sudden operation.

The above-described embodiments 1 to 16 can be implemented in any suitable combination.

Prediction and planning unit 100, DNN 101, reservoir 102, annealing machine 103, server 104, wireless interface 105, sensor 200, output unit 300, simultaneous multi-connection pipeline 1100, data flow controller 1110, time-space organizer 1120, edge 1200, overall control unit 1240

Claims (15)

A processing unit including at least one processing device selected from a function approximator and an annealing machine;
A data flow controller is provided for receiving data transmitted from a sensor and transmitting the data to the processing unit.
The data flow controller rearranges the data to be transmitted to the processing unit based on a transmission timing on the sensor side;
the data flow controller changes a ratio of data to be rearranged when rearranging the data to be transmitted to the processing unit based on a transmission timing on the sensor side;
The accuracy and speed of the processing by the processing unit can be evaluated by changing the proportion of the data to be rearranged.
Information processing system. The data received by the data flow controller includes at least one of data obtained from different sensors, data passed through different paths, and data by different communication means.
2. The information processing system according to claim 1. an orchestrator that configures the data flow controller in response to the configuration of the processing device;
3. The information processing system according to claim 2. a first measurement unit that broadcasts the time of its own reference clock to the sensor side;
4. The information processing system according to claim 3. a second measurement unit that broadcasts its own location information to the sensor side;
5. The information processing system according to claim 4. a delay time data table storing a transmission delay time from the sensor side;

a time-space organizer for labeling the data with the transmission timing and location of the sensor side based on the transmission timing of the sensor side obtained by subtracting the transmission delay time of the delay time data table from the arrival time of the data transmitted from the sensor side and the location information added to the data transmitted from the sensor side;
6. An information processing system according to claim 5. the data flow controller executes both a control for rearranging the data to be transmitted to the processing unit and a control for canceling the rearrangement control;
2. The information processing system according to claim 1. The data flow controller controls whether to rearrange the data based on the sensor type, the source, the line, or an arbitrary grouping when changing the proportion of data to be rearranged.
2. The information processing system according to claim 1 . The data flow controller changes the ratio of data to be rearranged so that the ratio increases monotonically.
2. The information processing system according to claim 1 . Performing at least one of feature extraction using a DNN or a reservoir as the processing device and future prediction using a reservoir as the processing device;
2. The information processing system according to claim 1. The processing unit includes a server, which generates an optimization problem based on the output of the DNN or the reservoir, converts the optimization problem into a model that can be processed by the annealing machine, sets the problem to the annealing machine, and reads out the solution after the problem is solved by the annealing machine.
11. The information processing system according to claim 10. The processing unit includes a parent device and a plurality of child devices,
the child device constitutes at least a part of at least one type of device selected from the function approximator and the annealing machine;
each of the parent device and the plurality of child devices includes a communication interface;
The communication interface is at least one selected from a wireless communication interface and a wired communication interface including an analog circuit;
data to be processed by the child device is transmitted from the parent device to at least one of the plurality of child devices;
an output of at least one node of the child device is transmitted to the parent device and at least one of the other child devices;
2. The information processing system according to claim 1. A plurality of the information processing systems according to claim 1 are provided,
The information processing system has a hierarchical structure including, in order from the sensor side, a first information processing system, a second information processing system, and a third information processing system.
Hierarchical edge systems. In the first information processing system, feature extraction is performed using a DNN or a reservoir as the processing device, and an emergency response is performed for a sudden operation.
The hierarchical edge system of claim 13. An information processing method for processing data transmitted from a sensor side by a processing unit including at least one processing device selected from a function approximator and an annealing machine,
A data flow controller receives data transmitted from the sensor side and transmits the data to the processing unit,
The data transmitted from the sensor side includes at least one of data obtained from different sensors, data via different routes, and data via different communication means;
The data flow controller rearranges the data to be transmitted to the processing unit based on a transmission timing on the sensor side;
the data flow controller changes a ratio of data to be rearranged when rearranging the data to be transmitted to the processing unit based on a transmission timing on the sensor side;
The accuracy and speed of the processing by the processing unit can be evaluated by changing the proportion of the data to be rearranged.
Information processing methods.

Images