JP6770922B2 - Data collection infrastructure - Google Patents

Description

The present invention relates to a data collection platform suitable for a gateway-intensive data collection system.

The Internet of Things (IoT) system, which collects and analyzes multiple sensor information according to the requirements of the application, is required for cloud-intensive type that aggregates and analyzes all sensor data on a cloud server, and for gateways and edge servers. There is a gateway-intensive type that collects data by aggregating it into granularity.

The latter gateway-intensive type has the advantage of saving network resources and storage resources as compared with the cloud-intensive type of the former. In this type of gateway-intensive IoT architecture, for example, a technique for optimizing the role of an edge server for the entire cloud has been proposed (see, for example, Non-Patent Document 1).

Norio Shiratori, Shinji Kitakami, Takuo Suganuma, Kenji Sugawara, Kaoru Shimamoto "Latest Trends in IoT Architecture" Journal of the Institute of Electronics, Information and Communication Engineers Vol. 100, No. 3, pp. 214-221, March 2017.

However, in the gateway-intensive type, it is necessary to design the module layout according to the application to be used in advance and arrange the modules based on the design. Therefore, when adding a new application or sensor, there is a problem that the work required for designing and relocating the module relocation is large and it is difficult to respond quickly.

The present invention has been made in view of the above circumstances, and an object of the present invention is to flexibly arrange modules on a data processing node according to an application to be arranged in a gateway-intensive data collection system. The purpose is to provide a data collection platform that can be carried out quickly.

One aspect of the present invention is an IoT platform in which one or more applications and a plurality of sensors are connected via a plurality of data processing nodes by a network, a requirement management unit that accepts application requirements, and each of the above data. The plurality of data are based on the resource management unit that manages the computational resources and network bandwidth information of the processing node as resource information, the application requirements received by the requirement management department, and the resource information managed by the resource management department. It includes a infrastructure management unit having an arrangement calculation unit that performs an arrangement calculation of a module that executes the application on the processing node, and an arrangement execution management unit that deploys a module to the data processing node according to the calculation result in the arrangement calculation unit.

According to the present invention, in a gateway-intensive data collection system, it is possible to flexibly and quickly arrange modules on a data processing node according to the application to be arranged.

The block diagram which shows the functional structure of the data collection platform which concerns on one Embodiment of this invention. The figure which shows the relationship between the processing flow which concerns on this embodiment, and various helper programs. The figure which illustrates the description method of the application requirement which concerns on this embodiment. The figure which illustrates the description method of the application requirement which concerns on this embodiment. The figure explaining the benefit of the application which concerns on the embodiment. The sequence diagram which shows the processing content when a new application is added to the IoT system which concerns on the same embodiment. The flowchart which shows the processing content of the arrangement calculation executed by the arrangement calculation part which concerns on the same embodiment. The figure which illustrates the module structure arranged by the arrangement calculation part which concerns on the same Embodiment, and the contents of the module structure creation program table which is created to obtain the structure. The figure which illustrates the configuration of the data processing node at the time of acquisition of the minimum module and the module configuration which concerns on the same embodiment. The figure which illustrates the configuration of another data processing node at the time of the minimum module acquisition processing which concerns on the same embodiment, and the module configuration. The figure which illustrates the configuration which shared the module at the time of the 1st module sharing process which concerns on the same embodiment, the sharing determination program table, and the upward compatibility sharing determination program table. The figure which shows the 1st example of the 1st module sharing process which concerns on the same embodiment. The figure which shows the 2nd example of the 1st module sharing process which concerns on the same embodiment. The figure which shows the 3rd example of the 1st module sharing process which concerns on the same embodiment. The figure which shows the 4th example of the 1st module sharing process which concerns on the same embodiment. The figure which illustrates the equivalent module table and the equivalent module column calculation program table used in the equivalent module expansion processing which concerns on the same embodiment. The figure which captures and explains the equivalent module expansion processing which concerns on the same embodiment. The figure which shows the example of the module configuration at the time of the 2nd module sharing process which concerns on the same embodiment. The figure which illustrates the module configuration and the data processing node configuration at the time of allocation processing to the actual node device which concerns on the same embodiment. The figure which illustrates each table of the node information, the module information, and the sensor information managed by the resource management part which concerns on the same embodiment. The figure which illustrates the allocation configuration of the network resource at the time of the allocation process to the real node device which concerns on the same embodiment. The figure which illustrates the vector of the link information and the resource utilization rate managed by the resource management part at the time of the accommodation admissibility judgment processing which concerns on the same embodiment. The figure which simplifies the processing process in the case of performing the arrangement calculation by the local search method which concerns on the same embodiment. The sequence diagram which shows the process flow of the required resource calculation by dynamic monitoring which concerns on the same embodiment. The figure which illustrates the uninterrupted module switching process which concerns on the same embodiment. The figure which illustrates the realization method of the module stop in an abnormal state which concerns on the same embodiment. The figure which illustrates the execution priority and the contents of module management by the arrangement execution management part which concerns on the same embodiment. The figure which shows the example of the chaining and equivalent conversion of the module which concerns on the same embodiment. The figure which shows the example of resource reduction by recombination of the intermediate data module which concerns on this embodiment. The figure which shows the 2nd example of resource reduction by recombination of the intermediate data module which concerns on the same embodiment. The figure which illustrates the conversion of the module configuration which concerns on the same embodiment. The figure which illustrates the conversion of the module configuration which concerns on the same embodiment.

Hereinafter, an embodiment in which the present invention is applied to the infrastructure management unit included in the data collection infrastructure of the IoT system will be described in detail with reference to the drawings.

FIG. 1 is a block diagram showing a functional configuration of a data collection platform of an IoT system. In the figure, the IoT system is composed of four layers of sensors 1, 1, ..., Gateway (IoT GW) 2, 2, ..., Edge servers 3, 3, ..., And cloud server 4, in order from the lower layer side. The case is shown as an example.

The middle layer gateways 2, 2, ... And the edge servers 3, 3, ... collect a plurality of data from the immediately lower layers based on the modules deployed by the placement execution management unit 15 described later as data processing nodes, respectively. , Send to the layer directly above.

The cloud server 4 on the uppermost layer collects and analyzes data from the edge servers 3, 3, ... Directly below the layer according to the assigned application 5.

The application 5 transmits information on required requirements in advance to the infrastructure management unit 10 that constitutes the data collection infrastructure, which is an IoT platform, via the requirement reception interface 6.

The infrastructure management unit 10 includes a database 11 that stores various programs and data to be executed in the infrastructure management unit 10, a requirements management unit 12, a resource management unit 13, an allocation calculation unit 14, and an arrangement execution management unit 15. ..

The requirements management unit 12 accepts the requirements required by the application 5 through the requirements reception interface 6.

The resource management unit 13 grasps the arrangement of gateways 2, 2, ... And edge servers 3, 3, ... As data processing nodes, the network bandwidth, the upper limit of computational resources, and the like.

The layout calculation unit 14 calculates the configuration and layout of the module that performs the data aggregation process based on the requirements of the plurality of applications.

The arrangement execution management unit 15 deploys the module to each actual data processing node based on the module configuration and arrangement calculated by the arrangement calculation unit 14.

FIG. 2 is a diagram showing the relationship between the processing flow mainly executed by the arrangement calculation unit 14 and various helper programs. The layout calculation unit 14 creates a module configuration that requires the module configuration creation program 21 from the requirements required for each application and the resource information of the resource management unit 13 (not shown here). These multiple module configurations are combined to form the initial module configuration.

With respect to this initial module configuration, the sharing determination program 22 performs a sharing determination of a module that executes the same process, and creates a necessary module configuration.

The equivalent module sequence calculation program 23 further examines the contents of the module configuration created in this way, and rearranges the module configuration into a more efficient module configuration.

Then, these module configurations are reconfigured by the program 24 for allocation, accommodation availability determination, and optimum configuration selection, and then deployed by the arrangement execution management unit 15 to each data processing node in the actual environment. Ru.

3 and 4 are diagrams illustrating a method of describing application requirements.
FIG. 3 describes an application requirement named "SensingApp1", specifying multiple data channels of a data channel CH1 named "data" and a data channel CH2 named "alert" as shown. There is.

In the data channel CH1, the module name (AVG) “AVG” as a single module and its parameters are described, and the priority information of the channel is described as “1”.

Similarly, in the data channel CH2, it is described that the priority information of the channel is "5". In this case, it is assumed that the higher the numerical value of the priority information, the higher the priority.

FIG. 4 describes an application requirement named "Trigger". In the region AR1 shown in the figure, the requirements related to the performance of Throughput and its parameters are described. Regarding this description, for example
"Benefit Function: function (t) ...
BenefitFunctionName: Myfunction ... "
It is also conceivable to describe as.

Next, the benefits of the application will be described.
Applications can describe throughput and latency performance requirements as benefits.

Both FIGS. 5 (A) and 5 (B) are models that describe the benefits of the application with respect to throughput. In FIG. 5A, when the throughput is x1 or less, it is “0 (zero)”, the benefit increases from y1 to y2 from x1 to x2, and when x2 or more, it becomes a constant value y2.
FIG. 5B shows a case where the parameter is increased to 3 points.

In this way, the application can describe the benefits with the parameter sequences x1, y1, x2, y2, ....

The application may also be configured to give a specific benefit program or the name of the program.

As the value of the benefit, for example, "1" is arranged as the minimum acceptable value, for example,
"Benefit:" function (t) {if (t <100) {return0}
else {return1}} ""
Or
"Benefit:" myOriginal BenefitFunction ""
It can be realized by the description like.

FIG. 5C is a model showing a case where the parameter related to performance is latency. Here, when the latency is x1 or less, the benefit decreases from y1 to y2 while the benefit is a constant value y1, x1 to x2, and when the latency is x2 or more, it becomes "0 (zero)".

In the case of latency, the smaller the latency, the higher the benefit, and as in the case of throughput, the application can describe the benefit with the parameter sequence x1, y1, x2, y2, ....

Hereinafter, specific operations of the above-described embodiment will be sequentially described.
FIG. 6 is a sequence diagram showing the processing contents when a new application is added in the IoT system. Initially, the application 5 transmits the requirement data in the format as described with reference to FIGS. 3 and 4 to the requirement management unit 12 of the infrastructure management unit 10 via the requirement reception interface 6 (). Step S101).

Upon receiving this, the requirement management unit 12 notifies the placement calculation unit 14 that the requirement has been changed due to the addition of the application (step S102).

In response to this notification, the placement calculation unit 14 requests the requirement management unit 12 to acquire a list of requirements (step S103), and receives the response information from the requirement management unit 12 (step S104).

At the same time, the allocation calculation unit 14 requests the resource management unit 13 to acquire the resource information (step S105), and receives the response information from the resource management unit 13 (step S106).

The placement calculation unit 14 executes the placement calculation process based on the list information of the requirements indicating the latest status and the resource information obtained in this way (step S200). The placement calculation unit 14 transmits to the requirement management unit 12 as a relocation possibility result notification whether or not a placement configuration satisfying the newly added requirement is obtained by this placement calculation (step S107).

Upon receiving the relocation permission / rejection result notification, the requirement management unit 12 responds to the application 5 via the requirement reception interface 6 with a response as to whether or not the requirement can be accepted in response to the notification.

Further, when the arrangement configuration satisfying the newly added requirement is obtained in the arrangement calculation, the arrangement calculation unit 14 further instructs the arrangement execution management unit 15 of the calculated arrangement (step S109).

According to the arrangement instruction from the arrangement calculation unit 14, the arrangement execution management unit 15 notifies the plurality of data processing nodes 1 and 2 of the arrangement instruction of the module to be operated on the node (step S110).

Specifically, the placement instruction notified here is composed of a set of three types of instructions: module start, module stop, and module setting change.

Next, the details of the operation procedure executed by the arrangement calculation unit 14 by the arrangement calculation S200 will be described.

FIG. 7 is a flowchart showing the processing contents by the arrangement calculation unit 14. At the beginning, the arrangement calculation unit 14 obtains the module arrangement of the minimum configuration that realizes the required channel from the request of the application. The requirements of all the applications are executed in the same manner, and the module configuration C1 obtained by adding all the results is obtained (step S201).

FIG. 8A is a diagram illustrating the module configuration C1 arranged by the arrangement calculation unit 14 at this time. The state in which the output data of the three sensors 1, 1, 1 are collected by the modules A, B, and B and collected by the channels CH1, CH2, and CH3 in the applications 1 and 2 is illustrated.

The process of step S201 will be described in detail. First, the arrangement calculation unit 14 describes the algorithm name described in the requirements of the application and the program name corresponding to it to create the minimum module configuration as a module configuration creation program table.

FIG. 8B is a diagram illustrating the contents of the module configuration creation program table created at this time. It is a correspondence table of the algorithm name and the program name, and the algorithm name described inside each channel of the application request is searched from the program table created above, and the corresponding program name is obtained. The program in which the program name is described here is stored in the database 11 in advance.

That is, each program is given the program name obtained as described above by adding the parameters in the corresponding channel in the application and the configuration of the data processing node managed by the resource management unit 13. It is stored in the database 11 in a preconfigured state so that the minimum module configuration can be acquired.

For example, application requirements as shown below
"{{
name: simple_time_series
Processing: {
AgorithmName: Time_Series_AVG,
Aggregation Unit: 3,
targetSensors: [“ID-1”]
} "
Suppose that is given.

FIG. 9 (A) illustrates the configuration of a data processing node when the data output by the sensor “ID-1” is collected by the node 1 and further combined with other data by the node 3 above the node 1. ..

The algorithm named "Time_Series_AVG", which is also shown in FIG. 8B, obtains the average value of the data at regular times when the time series data is output to the sensor.

The periodic time can be set as a parameter, and in this example, "3 [seconds]" described in the place of "Aggregation Unit" is a section for calculating the average of the time series data.

A module configuration is created and returned by giving the data and data node configuration of the channel portion within the above application requirements to the program with the program name "GEN_MODE_TIME_SERIES_AVG" shown in FIG. 8 (B).

As the module configuration is illustrated in FIG. 9B, in the returned module configuration, the connection relationship starts from the sensor, passes through one or more modules above, and reaches the node specified by the channel name. It is expressed as information.

As a different example, for example, the application requirements shown below
"{{
name: Tokyo_temperature,
Processing: {
AlgorithmName: AVG_LATEST,
Aggregation Unit: 60,
targetSensorsByType:
"& ((Type = temperature) (region = Tokyo)"
} "
Suppose that is given.

FIG. 10A exemplifies the configuration of the data processing node passed at this time. The algorithm name "AVG_LATEST" is used to obtain the average value of the latest values of a plurality of sensors, and the average value is calculated from the latest value of each sensor every "60 [seconds]" specified in the "Application Unit". Indicates that it will be delivered to the application.

A module configuration is created and returned by giving the data and data node configuration of the channel portion within the above application requirements to the program with the program name "GEN_MODULE_AVG_LATEST". The module configuration is illustrated in FIG. 10 (B).

In the example shown in FIG. 9 above, the sensor ID is directly specified, but here the requirements are specified by the sensor attributes. Here, it is described based on the format of "LDAP Filter", and the sensor whose sensor type is "Temperature" and whose region is located in "Tokyo" is specified. are doing.

By enabling such designation, the application does not need to specify the application for the sensor that is added daily in the IoT system, and the module corresponds to the configuration of the newly added sensor. There will be an advantage that you can actually see the placement calculation of.

When the configuration of the minimum module in step S201 of FIG. 7 is completed, the arrangement calculation unit 14 then executes the module sharing process as the first module sharing process if the module sharing process can be performed, and acquires the module configuration C2 (as it is). Step S202).

FIG. 11A illustrates a module configuration C2 in which two modules B are shared with respect to the module configuration shown in FIG. 8A.

In the process of step S202, by sharing the module being configured,
Search for cases where (1) the number of modules can be reduced without changing the processing content, and (2) cases where the number of modules is not reduced but the resources used for processing can be reduced.

Pay attention to the two modules in the configuration, check if they can be shared, and if they can be shared, replace them. Further, the process of sharing if there is a module that can be shared is repeatedly executed until it is determined that further sharing is impossible.

If you classify the cases that can be shared again here,
(1) Same type and same setting
(2) Different settings of the same type
(3) Sharing with upward compatible modules
(4) Mutual upward compatible modules
Will be.

The same type and same setting in (1) above indicates a case where the same input is received, the modules of the same type, and the setting is the same.

The same type different setting in (2) above indicates a case where the same input is received, the modules of the same type have different settings. In this case, it is possible to convert by the program by passing two different types of settings using the program corresponding to the program name corresponding to the module name defined in the shared program table shown in FIG. 11 (B). A set of structures is returned, so replace it with the returned structure.

If multiple types of substitutions are possible, multiple results are returned. If there are multiple structures, processing is continued for each structure. If sharing is not possible, or if it exists but is inefficient, an empty set is returned assuming that sharing is not performed.

For sharing with the upward compatible module in (3) above, refer to the upward compatible sharing judgment program table shown in FIG. 11 (C), and refer to the upward compatible module name corresponding to the module name and the upward compatible sharing judgment. Whether or not there is an upward compatible module of the module can be determined by whether or not there is a program name.

If the two modules of interest have this upward compatibility relationship, they may be shared. By inquiring the module name and setting related to sharing as parameters to the upward compatibility sharing judgment program referred to in this upward compatibility sharing judgment program table, a set of structures after sharing is returned. If there are multiple sets of returned structures, processing is continued for each structure.

For example, among a plurality of modules, the module "MAX" that calculates the maximum value every 5 [seconds] in the upward compatibility sharing determination program table shown in FIG. 11 (C) and the statistics for each 5 [seconds] If there is a module "STAT" that calculates, the latter upward compatible module "STAT" can be shared.

The mutual upward compatible module in (4) above indicates a case where there is a possibility that the upward compatible module can be shared by the upward compatible module when there is an upward compatible module common to the two modules of interest. Similar to the case of sharing with the upward compatible module in (3) above, the upward compatible sharing determination program searched from the evaluation is inquired to obtain a replaceable module configuration group.

FIG. 12 is a diagram showing a first example of the first module sharing process, and shows the case of the same type and the same setting in (1) above. When there is a module configuration as shown in FIG. 12 (A), paying attention to the module M102 and the module M103, since the same input, the same type, and the same setting are used, as shown in FIG. 12 (B). The existence of the module M103 can be abolished as it can be shared with the module M102.

FIG. 13 is a diagram showing a second example of the first module sharing process, and shows the case of the same type and different setting in (2) above. When there is a module configuration as shown in FIG. 13A, paying attention to the module M202 and the module M203, only the settings are different for the same input and the same type. Here, by inquiring about the program "SHARE_MAX" shown in FIG. 11 (B), the configuration as shown in the module M206 in FIG. 13 (B) is returned.

That is, although the number of modules due to the sharing process has not changed, the number of resources consumed by the system can be suppressed.

For example, when the sensor ID-1 in the figure outputs 1000 measurement data per second, the amount of messages in the module M204 and the module M205 is 2000 per second in the configuration shown in FIG. 13A.

On the other hand, in the post-sharing configuration shown in FIG. 13B, the total message amount per second of the module M207 and the module M206 is 1001, which can be substantially halved.

In terms of the amount of calculation, the module M202 and the module M203 before sharing performed 2000 times per second, whereas the module M208 and M210 after sharing performed 1001 times per second, and the amount of calculation was also approximately. Can be halved.

The above program "SHARE_MAX" is premised on being a program written by a developer who is proficient in the MAX module.

FIG. 14 is a diagram showing a third example of the first module sharing process, and shows the case of sharing with the upward compatible module of (3) above. When there is a module configuration as shown in FIG. 14A, focusing on the module M302 and the module M303, it corresponds to sharing with an upward compatible module. Here, by inquiring about the program "UP_SHARE_CHECK_STAT" shown in FIG. 11 (C), the configuration as shown in the module M309 in FIG. 14 (B) is returned.

Here, sharing increases the number of modules, but reduces the number of resources consumed by the system.

The "STAT" shown in the module M302 and the module M308 in the figure is a module for calculating the statistic of the input data, and in the JSON (Javascript (registered trademark) Object Notification) description format, for example.
"{{
“Max”: 120,
“Min”: 12,
"Avg": 60,
"N": 1000
"Sum": 64000,
"Sum2": 5888800
} "
(However, max is the maximum value, min is the minimum value, avg is the average value, n is the number of data, sum is the total data, and sum2 is the sum of squares of the data.)
The description content is as follows.

Further, the module M313 "EXTRAC T" in FIG. 14B is a module for extracting data designated as "param", and outputs a value 120 of "max" when it receives the above-mentioned format.

For example, when the sensor ID-1 in the figure outputs 1000 measurement data per second, the amount of messages in the module M304 and the module M305 is 2000 per second in the configuration shown in FIG. 14A.

On the other hand, in the post-sharing configuration shown in FIG. 14B, the total number of messages per second for the module M307, the module M311 and the module M312 is 1002, which can be substantially halved.

In terms of the amount of calculation, the module M202 and the module M203 before sharing performed 2000 times per second, whereas the modules M308, M311 and M312 after sharing performed 1002 times per second, and the calculation was performed. The amount can be halved.

FIG. 15 is a diagram showing a fourth example of the first module sharing process, and shows the case of the mutual upward compatible module of the above (4). When there is a module configuration as shown in FIG. 15A, paying attention to the module M402 and the module M403, they can be shared by mutual upward compatible modules. Here, by inquiring about the program "UP_SHARE_CHECK_STAT" shown in FIG. 11 (C), the configuration as shown in the module M412 in FIG. 15 (B) is returned.

Here, sharing increases the number of modules, but reduces the number of resources consumed by the system.

Here, the case of the module configuration for obtaining simple statistics has been described as in the case shown in FIG. 14, but the present embodiment is not limited to this, and examples of data include images, values of composite sensors, and the like. It is assumed that various things such as data processed for machine learning can be handled on the IoT system.

When the first module sharing process in step S202 of FIG. 7 is completed, the arrangement calculation unit 14 then executes a possible replacement process for each module included in the module configuration C2 as an equivalent module expansion process. It is stored as a set C3 of the module configuration (step S203).

That is, the modules included in the module configuration C2 are replaced by finding an equivalent module sequence. Here, the equivalent module indicates that the same processing can be performed in terms of data processing even if they are replaced. If there are multiple replacement targets for each module, replace all those patterns.

For example, in the module configuration C2 shown in FIG. 11 (A) above, when there are two modules, module A and module B, if there are two replacement patterns for each, nine types of module configurations can be configured as module configurations. Output as a set C3.

FIG. 17 is a diagram for explaining this point again. It is assumed that there are modules A1 and A2 as equivalent modules of module A, and there are modules B1 and B2 as equivalent modules of module B. When the state of the module configuration in FIG. 17 is described as [A, B], the equivalent module expansion process in step S203 described [A, B] [A, B1] [A, B2] [A1, B] [ A total of nine types of module configurations, A1, B1], [A1, B2], [A2, B], [A2, B1], and [A2, B2], are output as replaceable equivalent module sequences.

There are two ways to actually find the equivalent set of module sequences.
The first method is a method of obtaining a set C3 of equivalent module sequences by searching the module sequences described in the equivalent module table prepared in advance in the database 11 in the infrastructure management unit 10 of FIG. 1.

FIG. 16A exemplifies the contents of this equivalent module table, and it is possible to calculate the equivalent module column by searching the database 11. In the table shown in FIG. 16A, the case where each module is set to one is described, but the number may be two or more.

Further, in FIG. 17, the case where the number of replacement modules is two has been described, but the number may be three or more. Even in a table in which the maximum number of replacement modules is n, a sequence of replacement modules of several k less than n can be expressed by leaving the value of the replacement module k + 1 blank.

In the second method, a program for obtaining an equivalent module sequence is registered in the equivalent module column calculation program table prepared in advance in the database 11 in the infrastructure management unit 10 of FIG. 1, and an inquiry using the table is performed. Is a method of obtaining a set C3 of equivalent module sequences by executing.

FIG. 16B exemplifies the contents of this equivalent module column calculation program table, and by passing the module name of the replacement source and the setting parameter of the replacement source to the program shown here, The program returns an array of replacement modules and their settings.

Further, the upper limit of the number of modules to be generated can be given to the program shown in the equivalent module column calculation program table. The purpose of the equivalent module expansion is to obtain the candidates for distributing the processing with respect to the number of data processing nodes passing in the vertical direction, and to decompose it into more basic logic to realize the above-mentioned module sharing. It is to raise the angle.

Therefore, even if it is expanded to a large number of equivalent modules with respect to the number of data processing nodes passing through in the vertical direction, a large utility cannot be obtained, and the result is that the calculation time is wasted. Therefore, it is considered sufficient to give about twice the number of data processing nodes arranged in the vertical direction as the upper limit of the modules to be generated.

It is assumed that the equivalent module sequence calculation program shown in FIG. 16B is a program described by a developer who is proficient in the properties of the corresponding modules.

When the equivalent module expansion process in step S203 of FIG. 7 is completed, the arrangement calculation unit 14 then performs the module sharing process for each element of the set C3 of the module configuration as the second module sharing process. If possible, it is carried out as it is and added to the set C4 of the module configuration (step S204).

The sharing process is executed by almost the same method as the sharing process described in step S202, but in step S202, the sharing process is performed according to the order found, but here, sharing is performed. The difference is that the order is also considered as a variation.

That is, among the modules included in the module configuration C2, a combination of sharable modules is searched, and a shared configuration is obtained for each candidate.

FIG. 18A shows an example of a specific module configuration. Here, the module configuration C2 has the partial module configurations S301 and S302, and the module M1 and the module M2 of the partial module configuration S301 are shared to form the module configuration S304, and the module M2 and the module M3 are shared to be the module configuration S303 and the module. It is assumed that M2 and module M3 are shared to obtain the module configuration S305.

This procedure is applied to another partial module configuration S302 in the module configuration C2 and the newly created module configurations S303, S304, and S305, and is repeatedly executed until the new module configuration cannot be created. A set S310 having a module configuration as shown in FIG. 18B is generated.

As the set C4 of the module configurations created in step S204, all the configurations included in the set S310 of the module configurations may be included, or the set C4 existing on the end side of the set S310 is further shared. Only the no configuration and 311, 312, S313, S314, S315 may be included in the set C4 of the module configuration.

These are determined by the balance between the cost required for the placement calculation by the placement calculation unit 14 and the degree to which the omission of the search for the optimum solution is allowed.

When the second module sharing process in step S204 of FIG. 7 is completed, the arrangement calculation unit 14 then executes the allocation process to the real node device (step S205).

Here, the arrangement calculation unit 14 allocates modules to each data processing node for each module configuration which is an element of the set C4 of the module configuration, and adds the configuration to the set C5. When there are a plurality of allocation methods, they are added to the set C5 as a plurality of elements.

More specifically, first, the data processing node is assigned to the configuration corresponding to the configuration of interest. FIG. 19A is an example showing two modules including module A and module B, which are the configurations of interest. "VL" in the figure is a prefix intended as a virtual link.

On the other hand, FIG. 19B illustrates the configuration of nodes 1 to 4 which are data processing nodes. "PL" in the figure is a prefix intended for a physical link.

Basically, there are as many allocation methods as there are products of the number of modules and the number of nodes, but some of them are obviously inefficient and some are difficult to execute. Exclude the method.

On the other hand, the data processing node also manages the amount of resources provided and the special functions they hold. It also manages hardware characteristics such as CPU type.

On the other hand, the module that processes data also manages hardware characteristics such as the minimum required resources, required special functions, and required CPU type.

Therefore, an assignable pattern is generated based on each information of these modules and the data processing node. For example, in each configuration shown in FIG.
Node 3 ← Module A and
Node 1 ← Module B
Pattern and
Node 3 ← Module A and
Node 2 ← Module B
A total of two types of patterns can be calculated.

In actually executing the allocation to the actual node device, the allocation calculation unit 14 is difficult to allocate by referring to the node information table managed by the resource management unit 13 as shown in FIG. 20 (A). Modules can be calculated.

The node cannot be arranged unless the CPU architecture is supported by the module, and it is similarly difficult to arrange the node when the resource required by the module exceeds the resource amount of the data processing node.

In one data processing node, a plurality of modules will operate due to sharing, but in this step S205, no consideration is given to other modules, and a single module occupies the resources of the data processing node. If so, determine whether or not it can be placed.

Further, the resource management unit 13 manages the module information shown in FIG. 20 (B) and the sensor information shown in FIG. 20 (C) together.

Regarding "number of processing steps required for 1 event", "number of required memories", "required storage amount", "data output frequency", "output data size" in this module information, and "event occurrence frequency" and "data size" in sensor information, Although it is described as a fixed value, statistical values such as minimum value, maximum value, mean value, standard deviation, etc. are described, and statistical judgment is made based on the description contents. It may be determined whether or not the allocation is possible.

As mentioned above, in addition to the configuration that estimates the required resource amount by searching the table, it is necessary for the helper program of the module, which is another target program created by the module developer. It may be realized by a mechanism to inquire about the amount of resources.

In that case, by passing the input event frequency as an argument, even if the required resource is a function of the input event frequency, it can be approximated with high accuracy.

Next, a case where network resources are allocated will be described.
Here, as shown in FIG.
Module A: Node 3 and
Module B: Node 1
The case of allocating to is described as an example.

By mapping the links between the modules determined in the previous step S204,
PL4 ← VL3
PL5 ← VL4-1
PL3 ← VL4-2
PL2 ← VL4-3 + VL2
PL1 ← VL1
Can be assigned. Here, the virtual link VL4 is represented as a connection of the virtual links VL4-1, VL4-2, and VL4-3.

When the allocation process to the actual node device in step S205 of FIG. 7 is completed, the arrangement calculation unit 14 then executes the accommodation availability determination process (step S206).

The arrangement calculation unit 14 determines whether or not the elements of the set C5 of the module configuration can be accommodated with respect to the amount of resources inquired to the resource management unit 13, and if it is determined that it is possible, accommodates the elements in the set C6. At that time, the utilization rate of each resource is also calculated.

FIG. 22A is a diagram illustrating a table of link information managed by the resource management unit 13. The elements of the set C5 of the module configuration include an allocation configuration for physical resources. For example
Node 1 ← Module B + Module C
Node 2 ← Module A
PL4 ← VL3
PL5 ← VL4-1
PL3 ← VL4-2
PL2 ← VL4-3 + VL2
PL1 ← VL1
The configuration is as follows. The above symbol "←" indicates that the module or virtual link on the right side is assigned to the physical link on the left side across this symbol.

Here, the placement calculation unit 14 confirms that each element of the physical resource exceeds the sum of the corresponding elements to be allocated.

For all physical resources, if the above is satisfied, it is determined that the resource can be allocated, and it is stored as an element of the set C6.

The above comparison can also be performed with the safety factor (k) set to "0.9". In this case,
Each element of physical resource * k <Σ (each corresponding element to be allocated)
It is confirmed that each element of the physical resource exceeds the sum of the corresponding elements to be allocated, depending on whether or not is satisfied. The safety factor (k) may be such that different vectors k = (k ₁ , k ₂ , ..., K _n ) are prepared in advance for each resource type.

Further, as described above, the resource utilization rate of each element is also calculated and held as a vector. The content to be held here is the following formula
Σ (corresponding element to be allocated / each element of physical resource * k)
It is represented by. FIG. 22B exemplifies the resource usage vector held by the placement calculation unit 14 at this time.

In addition, by performing the calculation including the billing model, it is possible to calculate the cost when operating the configuration for a certain period, for example, one month. Information on the utilization rate and cost of the above resources is also stored as additional information when storing in the set C6.

By carrying out the above processing for all the elements included in the set C5, a set C6 having a new module configuration is created.

If the performance requirement is described in the application requirement, confirmation of conformity with the performance requirement in the application requirement is also performed as a part of the above-mentioned accommodation availability determination process.

In this case, the throughput and latency can be calculated from the number of processing steps of each module, the number of operable steps per unit time of the case data processing node, and the bandwidth of each network with respect to the actually assigned configuration.

This compares the benefits of each application. If there is an application with a benefit of less than 1, it is determined that the configuration is not suitable.

Further, the benefit of the application is, for example, a vector sequence consisting of vector values "1.1", "1.2", and "1.5" corresponding to the vectors "b _app1 ", "b _app2 ", and "b _appn ". By keeping it as information attached to the configuration, it may be used in a later step.

When the accommodation / non-accommodation determination process in step S206 of FIG. 7 is completed, the arrangement calculation unit 14 finally executes the optimum configuration selection process (step S207), and finally ends the arrangement calculation process.

More specifically, the arrangement calculation unit 14 determines the optimum configuration from the elements of the set C6, the vector of the resource usage rate added to the set C6, and the cost information such as monthly.

The methods for this are "1. Low average resource usage rate", "2. Low average resource usage rate with different weights according to resource type", and "3. Low maximum resource usage rate". It is conceivable to judge by either "4. The one with the lowest monthly cost" or "5. The one with the highest benefit of the application".

Hereinafter, the objective function will be illustrated with respect to each of the methods "1." to "3."

In the above "1. The average resource usage rate is low", for example, the following equation is used as the objective function.
_{avg (r 1, r 2,} ‥‥ r n)
Is used.

In the above "2. Those with a low average resource usage rate with different weights according to the resource type", the objective function
_{a = (r 1, r 2} , ‥‥ r n), a and? a _n = 1 and becomes a multiplied to r,
a · r = a ₁ × r ₁ ＋ a ₂ × r ₂ ＋ ‥‥‥ ＋ an _n × r _n )
In the above "3. The one with the lowest maximum resource usage rate", the objective function is added to the safety factor of the entire system.
_{max (r 1, r 2,} ‥‥ r n)
Is used.

Further, the selection process of the optimum configuration in step S207 can be easily realized by making a judgment by combining a plurality of the methods shown in "1." to "5.".

In the method of the arrangement calculation by the arrangement calculation unit 14 described in FIG. 7 and below, the method of listing all the possible configurations and then selecting the best one from them was adopted.

On the other hand, an algorithm called a local search method is known, and a method applying this local search method can also obtain an optimal arrangement configuration.

FIG. 23 shows the processing process in a simplified manner. Starting from the initial configuration (S501) obtained in the first module sharing process of step S202, the steps of the equivalent module expansion process (E) and the shared modularization process (S) are advanced one by one. , Increase the module configuration.

With respect to the configuration, allocation processing to the actual node device and accommodation / non-accommodation determination processing are performed, and the calculation by the above-mentioned objective function is appropriately executed to obtain an evaluation for the configuration.

From the configuration with the highest value among those evaluation values, further proceed with the expansion process (E) of the equivalent module and the shared modularization process (S), or if there is no case to proceed, the previous one. The process of returning to the node or allowing the objective function to be deteriorated to some extent and reselecting the node configuration to be searched from all the nodes is sequentially executed.

By repeating these procedures a certain number of times, the best number of searches can be obtained. As for the degree of repetition described above, when a predetermined number of times are executed, when the improvement of the objective function converges, when all possible configurations are searched, a temperature that allows a worsening transition of the objective function called annealing method. Either of the cases where the judgment is made using parameters can be selected.

Further, in the local search method shown in FIG. 23, when a configuration that can be assigned once is obtained, it is better to deploy the module to the data processing node once with the configuration and start providing the function, and then in parallel. It is also possible to search for a configuration and change to a better configuration when a specific time has passed.

Further, in step S107 of FIG. 6, the requirements management unit 12 that has received the notification of the result indicating whether or not the relocation is possible from the arrangement calculation unit 14 has insufficient resources for the content of the notification, and there is a module configuration to be allocated. If not, the following steps may be performed.

That is, the requirement management unit 12 searches for whether or not a new application can be accepted by stopping one of the plurality of application requirements that is less important. Among the new application requirements, the application requirement having the highest priority is left, other application requirements are excluded, and then the placement calculation unit 14 is requested to perform the placement calculation again. The lowest priority that takes this requirement into account is referred to as the "executable priority".

For example, existing application requirements and their priorities
10: App_Critical
7: App_Important
4: App_soso
1: App_just_test
And the new application requirements and their priorities
8: App_Very_Important
5: App_just_for_fun
If this is the case, the application requirement of "8" on the new side is selected as the "executable priority".

In this state, the placement calculation unit 14 performs the placement calculation again, and if a module configuration capable of executing the application requirement cannot be obtained, the requirement management unit 12 informs the application 5 in step S108 according to the content of the notification. In response, we will notify you that it is not possible to answer whether or not the requirements are accepted.

Further, as a result of the placement calculation again, when a module configuration capable of executing the application requirement is obtained, the placement calculation unit 14 decrements the executable priority by 1 and then executes the placement calculation again. Is repeated until no executable module configuration is obtained.

If it can be determined that all the new requirements indicated by the application are feasible, a notification that it is entirely acceptable, and if it can be determined that it is partially feasible, it is partially acceptable. Notify as an answer.

Further, although not described in the above embodiment, in the data processing node, a small agent that manages a resource and a module operating is operated as, for example, a resource management unit, and the consumption measured by the resource management unit is used. By updating and setting the amount of resources in the module information managed by the resource management unit 13 using the resources, it is possible to calculate the required resources by dynamic monitoring.

FIG. 24 illustrates a sequence diagram at this time. For example, the resource utilization measurement unit of the data processing node 1 notifies the arrangement execution management unit 15 of the resource utilization information at that time at a preset timing.

Upon receiving this, the placement execution management unit 15 transfers the notification to the resource management unit 13, and the resource management unit 13 executes a process of updating the amount of resources required for the module configuration.

It should be noted that the above processing is executed when only one module is executed in the data processing node.

It is also conceivable to add and set a "flag measured with high accuracy" to the "module information managed by the resource management unit 13" of the module. In that case, if multiple modules are executed in one data processing node, but only one module is not given the "highly accurate measurement flag", the whole in the data processing node. The difference obtained by subtracting the resource amount of the module to which the "highly accurate measurement flag" is given from the usage amount of is regarded as the resource amount of the module to which the above flag is not given, and is updated. You can also.

In addition, when switching modules, the modules before and after the switch are arranged at the same time, and the modules are switched based on the time and sequence number assigned to the sensor data, so that the sensor data is not lost and more efficient. It is possible to switch to a standard module.

FIG. 25 is a diagram illustrating a switching process of this module.
The figure illustrates a case where the sensor data from the sensor ID-0 is first input to the channel 0 via the data processing node A and then via the data processing node B after switching. Time stamps and sequence numbers are stored in the sensor data.

In FIG. 25A, the arrangement execution management unit 15 tells channel 0 to ignore the input from the data processing node B until instructed to do so.

After that, as shown in FIG. 25 (B), the data processing node B is deployed, and the time stamp and the sequence number in the data from the most recently received sensor ID-0 are also acquired from the data processing node B.

Here, the arrangement execution management unit 15 notifies the channel 0 of the time stamp and the sequence number in the sensor data after a time having a certain margin, for example, about 1 [second] to 5 [second]. Therefore, the data input from the data processing node A beyond the time stamp and the sequence number is ignored, and the data input from the data processing node B is set to be used.

In this way, it is possible to switch the arranged modules without interrupting the input of the sensor data.

Next, a case where the module is stopped in an abnormal state will be described.
Assuming that each data processing node has a resource utilization measurement unit as described above, does the resource utilization measurement unit exceed the accommodation upper limit allowed by the node, for example, 80 [%] of the resource capacity? It monitors whether or not, and if the upper limit is exceeded, the module of the application to be used is stopped and the module with lower priority is stopped so that the application with higher priority is not affected.

FIG. 26 is a diagram illustrating a method of realizing module stoppage in an abnormal state.
As shown in FIG. 26 (A), in the data processing node, the resource utilization measurement unit M601 monitors resources in the node such as a CPU and a memory. When the resource usage rate exceeds a preset value, for example, 80 [%], as shown in FIG. 26B, for example, the current resource is sent from the data processing node 1 to the placement execution management unit 15. Notifies that the resource has been exhausted along with the usage status of (step S501).

The placement execution management unit 15 executes the execution priority calculation and the resetting process in response to this notification (step S502), and then transmits a module stop command to each data processing node 1 and 2 (step S503). , S504).

On the other hand, the placement execution management unit 15 transmits a change notification of the executable priority to the requirement management unit 12 (step S505). Upon receiving this, the requirements management unit 12 notifies the application 5 of the stop (step S506).

There are several methods for the placement execution management unit 15 to calculate the executable priority. For example
1. 1. A method in which the priority is set to 1 higher than the priority set in the module with the lowest priority that operates on the data processing node.
2. A method of calculating the theoretical resource usage rate of each resource when the executable priority is changed in advance, and selecting the executable priority so that the catch of the resource can be eliminated.
And so on.

FIG. 27 is a diagram illustrating the execution priority and the contents of module management by the arrangement execution management unit 15.
As shown in the figure, the placement execution management unit 15 manages a table consisting of a "module name", a "highest priority of the corresponding application", and an "execution state" for each data processing node, and manages a table of applications related to the module. Keep the one with the highest priority.

When the executable priority of the entire system is changed, the deployment execution management unit 15 starts the module whose "highest priority of the corresponding application" in the above table is equal to or higher than the "executable priority". Instruct each data processing node.

Next, the process of changing the safety factor will be described.
The layout calculation unit 14 sets a safety factor for each data processing node and each link, and plans the layout of the modules so as not to exceed the product of the total resource capacity multiplied by the safety factor.

In addition, it is possible to reset the safety factor during the actual execution of the application. In this case, if you want to change the safety factor, consider whether there is an application that cannot operate, and if there is such an application, notify the system administrator to that effect. Then, after the operation in which the presented contents are accepted is performed, the module related to the application is stopped, and the process related to the change setting of the safety factor is actually executed.

After changing the safety factor, the executable priority can be calculated by executing the placement calculation by the same procedure as the application selection process based on the executable priority described above.

Therefore, select an application that is less than or equal to this executable priority value and present it to the system administrator for confirmation. When confirmation is obtained by the system administrator, the above safety factor and the executable priority are changed, and the module is stopped accordingly.

As the timing to decide the arrangement of the module configuration, the case of adding a new application and the case of changing the safety factor have been described as an example, but in addition to these, when a data processing node is added, data processing is performed. If the node is a virtualized computer and the allocated resources change, the network capacity changes, the application requirements change, or the sensor installation changes. It can also be received in the same manner at each of these timings.

Further, as described with reference to FIG. 25, when the data processing nodes are switched without interruption, the number of modules to be temporarily operated may increase. In that case, the above-mentioned change setting of the safety factor is also temporarily implemented, the module related to the application with low priority is stopped, and then the above-mentioned uninterrupted switching of the data processing node is performed to switch. By setting the change to restore the safety factor again later, it is possible to switch modules without interruption even for high-priority applications.

Hereinafter, with reference to FIGS. 28 to 32, the operation in the case of rearranging the module configuration will be described by summarizing the explanations so far.

FIG. 28 shows an example of module chaining and equivalent exchange. The module "data processing 1" that performs statistics every 6 [seconds] shown in FIG. 28 (A) is the module "data processing 2" that performs statistics every 2 [seconds] shown in FIG. 28 (B). , And even if it is replaced with a two-stage column (chain) of the module "data processing 3" that performs statistics in units of three data, equivalent exchange is established.

Consider a case where the arrangement is changed from the module configuration shown in FIG. 28 (A) to the module configuration shown in FIG. 28 (B).

FIG. 29 shows an example of resource reduction by rearranging the intermediate data module.
First, as shown in FIG. 29 (A), application 1 that wants statistical information every 6 [seconds] is operating as an existing application, and there is statistical information every 4 [seconds] of the same sensor. It is assumed that a new application 2 is started.

By the above-mentioned "acquisition of minimum module" process, the arrangement plan as shown in FIG. 29 (A) is executed. Further, when the method described in "Application of local search method" described with reference to FIG. 23 is carried out, and when it takes time to acquire the configuration of FIG. 29 (B) described later. In the module configuration shown in FIG. 29 (A), there may be a case where the actual operation is started for the time being.

On the other hand, as shown in FIG. 29 (B), by converting to a module configuration equivalent to the module configuration of FIG. 29 (A) and further sharing the module configuration, the sensor (maximum commitment). (Number) Module "Data processing 3" that performs statistics every 2 [seconds], module "Data processing 4" that performs statistics in units of 3 data, and module "Data processing 4" that performs statistics in units of 2 data A desired module configuration can be obtained by substituting with the configuration according to "Data processing 5".

For example, when there are 100 data from the sensor per second, the load on the data processing node can be significantly reduced by changing to the module configuration as shown in FIG. 29 (B).

Further, the processing data having the same contents as that shown in FIG. 29A is passed to the applications 1 and 2.

Next, FIG. 30 shows a second example of resource reduction by rearranging the intermediate data module, which replaces the method described in FIG. 29.

FIG. 30A calculates and provides statistical information from sensors arranged nationwide for application 1. It is assumed that the application 2 that wants the statistical information about a specific area is newly added there.

FIG. 30B shows a case where the module configuration is equivalent to that of FIG. 29A by the arrangement conversion by the “acquisition of minimum module” process.

For this module configuration, the module "Data Processing 2" that wants statistical information of a specific area corresponding to the area where multiple sensors are installed, and the module "Data" that processes the statistical information of other areas. After changing the connection configuration between the sensor / module to "Process 1", the module configuration is such that the module "Data Process 3" that collects the statistical information and acquires the statistical information of the whole country is arranged. By performing the arrangement calculation as described above, it is possible to convert to a module configuration as shown in FIG. 30 (C).

In FIG. 30, the case where the number of sensors is four is illustrated in order to simplify the explanation. However, when there are about 100 sensors, for example, the module configuration shown in FIG. 30 (B) is shown. It can be easily understood that the module configuration shown in FIG. 30 (C) can significantly reduce the amount of resources used.

Next, an example of module conversion will be described with reference to FIG.

For example, when face detection processing is performed from camera images deployed in various places, the degree of similarity with the face image to be compared is calculated, and it is determined that the degree of similarity is higher than a certain value. Let's take an application that notifies the user as an example.

FIG. 31 (A) shows that the network from the camera to the server is in the rate-determining stage when the similarity comparison (step E101) is performed on the image data as the sensor data obtained from the camera which is the sensor. There is a possibility that

In that case, as shown in FIG. 31 (B), relatively lightweight preprocessing, specifically face detection processing and face region cutting processing (step E102), are performed on the gateway 2, and the amount of data and data After reducing the frequency, the module configuration is converted so that the similarity comparison process (step E103) by the upper server is performed.

By converting the module configuration in this way, it is possible to reduce the amount of transfer on the network and reduce the amount of resources used as a whole.

It is assumed that the similarity comparison module is executed in one instance on the cloud. If the execution processing for one data input is large, the throughput may not meet the requirements of the application.

FIG. 32 (A) shows the same module configuration as in FIG. 31 (B), and shows a case where there is only one module configuration.

With respect to such a configuration, in FIG. 32B, the similarity comparison process (step E103) is divided into a plurality of data processing nodes, and the distribution process (step E104) is arranged in front of the plurality of data processing nodes. An example is shown in which the result aggregation process (step E105) is arranged in the stage and the similarity comparison process (step E103) is distributed to a plurality of nodes to obtain the possibility of improving the throughput.

In the module configuration shown in FIG. 32B, it is considered that the latency increases because the number of passing modules increases.

According to the above embodiment, data processing can be performed efficiently by appropriately distributing the processing to vertically distributed processing nodes on the network instead of collecting all the data in the cloud.

In addition, since the module layout design is calculated mechanically by the system, not by the developer or operator, the corresponding operation becomes smaller, and the deployment of new applications can be speeded up and timely.

This makes it easy to run multiple applications on the same resource.

Furthermore, by writing the application requirements, the administrator does not need to design and implement the module layout, and the management cost can be suppressed and the response can be made quickly.

In addition, when the traffic increases or an abnormal process occurs, the operation of the application with high priority can be continued by stopping the module related to the application with low priority.

In the above description, although the description has been made using an example of a specific module, the present invention does not limit the contents of the module and the like, and a new data processing module and a new algorithm described in the application requirements. Even if is introduced, it will be possible if a "module configuration creation program", "sharing determination program", and "equivalent module column calculation program" that can act on these are provided.

A wide variety of data processing modules and algorithms can be created according to the demand from the application side. Further, for those who know the mechanism, it is easy to create a helper program (the above-mentioned "module configuration creation program", "sharing determination program", "equivalent module string calculation program").

It goes without saying that the present invention is still applicable as described above even in the situation where many data processing modules are added.

In addition, the present invention is not limited to the above embodiment, and can be variously modified at the implementation stage without departing from the gist thereof. In addition, each embodiment may be carried out in combination as appropriate, in which case the combined effect can be obtained. Further, the above-described embodiment includes various inventions, and various inventions can be extracted by a combination selected from a plurality of disclosed constituent requirements. For example, even if some constituent requirements are deleted from all the constituent requirements shown in the embodiment, if the problem can be solved and the effect is obtained, the configuration in which the constituent requirements are deleted can be extracted as an invention.

1 ... Sensor, 2 ... Gateway (IoT GW), 3 ... Edge server, 4 ... Cloud server, 5 ... Application, 6 ... Requirements reception interface (I / F), 10 ... Infrastructure management department, 11 ... Database, 12 ... Requirements management department, 13 ... resource management department, 14 ... placement calculation department, 15 ... placement execution management department, 21 ... module configuration creation program, 22 ... sharing judgment program, 23 ... equivalent module column calculation program, 24 ... allocation, accommodation A program for determining availability and selecting the optimum configuration.

Claims (7)

An IoT platform in which one or more applications and multiple sensors are connected by a network via multiple data processing nodes.
Requirements management department that accepts application requirements and
The resource management department that manages the computational resources and network bandwidth information of each of the above data processing nodes as resource information,
Based on the requirements of the application received by the requirement management unit and the resource information managed by the resource management unit, the layout calculation unit that performs the layout calculation of the module that executes the application on the plurality of data processing nodes, and the layout calculation unit.
A data collection platform including a platform management unit that has a layout execution management unit that deploys modules to data processing nodes according to the calculation results of the layout calculation unit. The requirements management unit receives the requirements of the newly added application and notifies the placement calculation unit.
The layout calculation unit determines whether or not a new application can be connected from the resource information managed by the resource management unit based on the requirements of the newly added application.
The requirement management unit responds to the new application with the determination result of whether or not the connection is possible by the placement calculation unit.
The data collection platform according to claim 1. The requirements of the application accepted by the requirements management department include priority information.
The placement calculation unit determines whether or not each application can be connected based on the priority information included in the requirements of the existing application and the priority information included in the requirements of the new application.
The data collection platform according to claim 2. The layout calculation unit determines whether or not the module in which the existing application is running can be shared, and if it is determined that the module can be shared, the module relocation calculation is performed, and the result executed by the module up to that point is used. The data collection platform according to claim 1, which continues the execution of the application. The data collection platform according to claim 1, wherein the allocation calculation unit performs allocation calculation of a module that executes an application by referring to resource information managed in advance by the resource management unit. The data processing node includes a resource measurement unit that measures resource information used in operation of the module and notifies the resource management unit.
The above resource management department updates and manages the resource information received from the data processing node.
The data collection platform according to claim 1. When switching modules during application execution, the placement execution management unit arranges the modules before and after the switching at the same time, and executes the module switching based on at least one of the time and sequence number information given to the data from the sensor. The data collection platform according to claim 1.