JP6900265B2 - Data analysis system and data analysis method - Google Patents

Description

The present invention relates to a data analysis system and a data analysis method.

In Patent Document 1, "data generated in the intermediate stage of analysis processing is efficiently saved and the intermediate data is reused", and "data generated in the intermediate stage of analysis is saved and saved data". The quantified feedback information for the data is accepted as the evaluation value, and the intermediate data for which no evaluation value is given is preferentially deleted, while the intermediate data for which a particularly high evaluation value is received is similar data. The intermediate data is automatically managed by background processing so that the analysis of the data to be compared and the analysis that is expected to be derived can be performed at high speed. "

Japanese Unexamined Patent Publication No. 2011-2911

In data analysis work using information processing equipment, the work is often carried out by repeating trial and error. Especially in data analysis targeting a large amount of data such as big data, the algorithms of the processing elements that make up the data analysis process, such as data collection, processing, classification, search, analysis, machine learning, and visualization ( In many cases, the work is carried out while adjusting the logic).

However, in such work, it is necessary to re-execute the entire data analysis process from the beginning every time some processing elements are changed, and until the result is obtained when the target data is huge or the process is complicated. It takes a long time, which is a factor that hinders the efficiency of data analysis work. Here, Patent Document 1 describes that intermediate data is reused, but does not disclose any mechanism based on the viewpoint of re-execution from the middle of a series of analysis processes.

An object of the present invention is to provide a data analysis system and a data analysis method capable of efficiently performing data analysis work.

One of the present invention is a data analysis system, which is deployed with a storage unit that stores analysis processing definition information including information defining an execution order of a plurality of processing elements constituting the data analysis processing. A processing execution unit that executes the processing element according to the analysis processing definition information by the material and stores data generated by each of the processing elements, and a material to be deployed by executing a build based on the source code of the processing element. The build execution unit that generates the data and the current deployment information that is the update information of the materials currently deployed in the processing execution unit are stored, the change of the source code is accepted, and the build is executed based on the changed source code. The processing execution unit includes a change management unit that generates the new material and generates change information that is information indicating the difference between the current deployment information and the update information of the new material. When executing the data analysis process, it is determined whether or not the processing element of the data analysis process has been changed based on the change information, and if it is determined that the processing element has been changed, the data analysis process is executed before the changed processing element. The execution of the processing element defined in the analysis processing definition information is omitted, and the processing element defined in the analysis processing definition information is generated in the past so as to be executed before the changed processing element. Using the data, processing is executed from the modified processing element.

In addition, the problems disclosed in the present application and the solutions thereof will be clarified by the column of the form for carrying out the invention and the drawings.

According to the present invention, data analysis work can be performed efficiently.

It is a figure which shows the schematic structure of the data analysis system. This is an example of an information processing device used to realize a data analysis system. It is a figure explaining the function which each of a source control apparatus, a build apparatus, and a change management apparatus has, and information (data) managed by these apparatus. It is a figure explaining the function provided by the change management apparatus and the execution apparatus, and the information (data) managed by these apparatus. This is an example of analysis process definition information. Currently an example of deployment information. This is an example of change information. It is a sequence diagram explaining the process performed in a data analysis system. It is a flowchart explaining the process performed by the execution device. It is a figure which shows an example of the data management form in a data lake. It is a figure which shows the relationship between the flow of the analysis process executed by the execution apparatus, and the data stored in the data lake by the analysis process. It is a figure which shows the relationship between the flow of the analysis process executed by the execution apparatus, and the data stored in the data lake by the analysis process. This is an example of the execution request instruction screen.

Hereinafter, embodiments will be described with reference to the drawings. In the following description, the same or similar configurations may be designated by the same reference numerals and duplicate description may be omitted.

FIG. 1 shows a schematic configuration of a data analysis system 1 described as an embodiment. The data analysis system 1 includes information processing devices belonging to each management segment of the server segment 2 and the user segment 3. The information processing device belonging to the server segment 2 and the information processing device belonging to the user segment 3 are connected to each other so as to be able to communicate with each other via the communication network 5. The communication network 5 is, for example, a LAN (Local Area Network), a WAN (Wide Area Network), the Internet (Internet), a dedicated line, or the like.

The user of the data analysis system 1 (a person who performs data analysis work, for example, a data scientist, a person who develops and maintains a program for processing related to data analysis (hereinafter referred to as data analysis processing), etc.) is a user segment 3. The user terminal 110 belonging to the user terminal 110 is used to access the information processing device belonging to the server segment 2.

As shown in the figure, the server segment 2 includes a plurality of information processing devices of the source control device 101, the build device 102, the change control device 103, and the execution device 104. In this example, the functions are distributed to a plurality of information processing devices in this way, but the plurality of functions may be realized in one or more common information processing devices.

FIG. 2 shows the source management device 101, the build device 102, the change management device 103, the execution device 104, and the user terminal 1 belonging to the user segment 3 belonging to the server segment 2.
10 is an example of an information processing device (hereinafter, referred to as an information processing device 10) used for realizing each of the above. As shown in the figure, the information processing device 10 includes a processor 11, a storage device 12, an input device 13, an output device 14, and a communication device 15. These are connected to each other so as to be able to communicate with each other via a communication means such as a bus. The configuration of all or part of the information processing device 10 may be realized by using virtual resources such as a cloud server in a cloud system.

The processor 11 is configured by using, for example, a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), or the like. For all or part of the functions of the source management device 101, the build device 102, the change management device 103, the execution device 104, and the user terminal 110, for example, the processor 11 reads and executes the program stored in the storage device 12. It is realized by doing. The information processing device 10 may have functions such as an operating system, a file system, a device driver, and a DBMS (DataBase Management System). The source repository 106, the archive repository 107, the data lake 108, the current deployment information 160, and the change information 170, which will be described later, can be realized or managed by using, for example, a DBMS.

The storage device 12 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a non-volatile semiconductor memory (Non-volatile memory), a hard disk drive, and an SS.
D (Solid State Drive), etc., which stores programs and data. Each information (data) stored in the storage device 12 is managed by, for example, a file system or a DBMS.

The input device 13 is a user interface that accepts external input, and is, for example, a keyboard, a mouse, a touch panel, or the like. The output device 14 is a user interface that provides various information such as processing progress and processing results to the outside, and is, for example, an image display device (liquid crystal monitor, LCD (Liquid Crystal Display), graphic card, etc.), a printing device, or the like. ..

The communication device 15 is a device that realizes communication with other devices, and is configured by using, for example, a NIC (Network Interface Card) or a wireless communication module. For example, the information processing device 10 may be configured to receive an external input via the communication device 15. Further, for example, the information processing device 10 may be configured to provide various information such as the processing progress and the processing result to the outside via the communication device 15.

FIG. 3 is a diagram for explaining the functions provided by each of the source control device 101, the build device 102, and the change management device 103 belonging to the server segment 2, and the information (data) managed by these devices.

The source management device 101 has a program management unit 201. The source control device 101 manages the source repository 106.

When the user accesses the source control device 101 via the user terminal 110 and commits (registers) the source code (hereinafter, also referred to as analysis logic) of the data analysis process, the program management unit 201 is created or updated by the user. Register the source code in the source repository 106. The program management unit 201 manages update information (revision information, version information, etc.) of the source code stored in the source repository 106. For example, the program management unit 201 assigns the same update information to a group of source codes committed at the same timing. The source code is described using, for example, the Java (registered trademark) language.

The build device 102 has a build execution unit 202. The build device 102 manages the archive repository 107. The archive repository 107 stores materials (hereinafter, also referred to as deployment target materials) to be deployed (production deployment) to the execution device 104, which are generated by executing the build. In this example, it is assumed that the above material is an application archive. An example of an application archive is the "jar" file in Java®.

The change management device 103 has a change management unit 203. The change management device 103 currently manages the deployment information 160 and the change information 170. The current deployment information 160 includes update information of each component (components of the application archive, for example, an execution module corresponding to the source code) included in the application archive currently deployed in the execution device 104. On the other hand, the change information 170 includes information indicating the difference between the update information of each component included in the application archive generated in the most recently executed build and the update information of each component of the current deployment information 160.

Upon receiving the "change information generation instruction" from the user terminal 110, the change management unit 203 receives information about the source code from the source control device 101 (program name of each source code (for example, program name 162 of FIG. 6 described later)). And update information of each source code (for example, update information 163 of FIG. 6 described later) is included. Hereinafter, it is also referred to as source information).

Further, the change management unit 203 receives information about the application archive from the build device 102 (information indicating which component is packaged in which application archive (for example, correspondence between the application archive name 161 and the program name 162 in FIG. 6 described later). Includes information indicating). Hereinafter referred to as build information).

The change management unit 203 has identified and identified the difference between the two by comparing the update information of each component contained in the application archive generated by the most recently executed build with the update information of the current deployment information 160. The result is output as change information 170. The details of the change information 170 will be described later.

When the change management unit 203 receives the "current deployment information update instruction" from the user terminal 110, the change management unit 203 updates the current deployment information 160 to the latest content based on the content of the change information 170.

FIG. 4 is a diagram for explaining the functions included in the change management device 103 and the execution device 104 belonging to the server segment 2, and the information (data) managed by these devices.

The execution device 104 (process execution unit) has each function of an execution request reception unit 301, a process execution control unit 302, a command call unit 303, an analysis logic execution unit 304, and a data input / output unit 305. Of these, the execution request receiving unit 301, the processing execution control unit 302, the command calling unit 303, and the data input / output unit 305 are provided to the user in advance as, for example, a platform system (basic system) for performing data analysis work.

As shown in the figure, the execution device 104 manages the data lake 108. The data lake 108 includes, for example, data to be analyzed (for example, data acquired from a website, data acquired from an SNS (Social Networking Service), data collected from a sensor, POS (point of sale) data, and the like. Questionnaire data, so-called big data, etc., hereinafter also referred to as analysis target data), data generated in each processing section of the data analysis processing executed by the execution device 104 (generated each time a processing element described later is executed) (generated. Hereinafter, intermediate data), data including information on the result of analysis processing (hereinafter, also referred to as result data), and the like are managed.

The data lake 108 stores, for example, the data to be analyzed in the format (native format) from which the data to be analyzed is acquired, and the data to be analyzed is required in response to a request from the data input / output unit 305. It is converted into data (structured data, semi-structured data, unstructured data, etc.) and provided to the data input / output unit 305. The above conversion is performed by using, for example, an ETL tool (ETL: Extract / Transform / Load).

The execution device 104 stores the analysis process definition information 306. In the analysis process definition information 306, information that specifies the execution order of a plurality of processing elements constituting the data analysis process is described. The analysis process definition information 306 is created by the user, for example, by using the setting function provided by the execution device 104 or the user terminal 110. In this example, the analysis process definition information 306 is assumed to be one of the materials to be deployed.

FIG. 5 shows an example of the analysis process definition information 306. As shown in the figure, in the analysis processing definition information 306, execution control statements of each processing element to be executed in order are described in the execution order. In this example, five processing elements 1 to processing element 5 are described in the order of execution. Each execution control statement includes a description of the file name of the execution command and the file to be executed (a "jar" format file in Java (registered trademark)).

Returning to FIG. 4, the execution request receiving unit 301 receives the execution request of the data analysis process from the user terminal 110. The analysis process definition information 306 may be stored in the execution device 104 in advance. For example, the execution request reception unit 301 receives the analysis process definition information 306 together with the execution request from the user terminal 110. You may try to remember it. When the execution request receiving unit 301 receives the execution request, the execution request receiving unit 301 instructs the processing execution control unit 302 to execute the data analysis process. Further, the execution request receiving unit 301 provides the user terminal 110 with the execution result of the data analysis process according to the execution instruction.

The process execution control unit 302 controls the execution of the data analysis process. The process execution control unit 302 executes the data analysis process according to the analysis process definition information 306 by the currently deployed application archive. When the process execution control unit 302 receives the execution instruction from the execution request reception unit 301, the process execution control unit 302 acquires the change information 170 from the change management unit 203 of the change management device 103, and inputs the contents of the change information 170 to the command calling unit 303.

The command calling unit 303 executes the data analysis process according to the analysis process definition information 306 (executes the processing element described in the analysis process definition information 306). The command calling unit 303 executes the analysis process in cooperation with the analysis logic execution unit 304. When the second execution mode described later is selected, the command calling unit 303 confirms whether or not each processing element has been changed (whether or not the application archive that realizes the processing element has been changed) based on the change information 170. When there is a change in the processing element, the execution of the optional processing element among the processing elements described in the analysis processing definition information 306 is omitted (skipped).

The analysis logic execution unit 304 executes an execution command (processing element) called from the command call unit 303. When the analysis logic execution unit 304 needs to access the data lake 108 (input or output of data) when executing the processing element, the analysis logic execution unit 304 proceeds with the processing in cooperation with the data input / output unit 305.

The data input / output unit 305 functions as, for example, a data access object, and accesses the data lake 108 in response to a request from the analysis logic execution unit 304.

FIG. 6 is an example of the above-mentioned current deployment information 160. As shown in the figure, the current deployment information 160 is composed of one or more records having each item of the application archive name 161 and the program name 162, and the update information 163.

Information that identifies the application archive is set in the application archive name 161. In this example, the file name of the "jar" file (hereinafter, also referred to as the application archive name) is set in the application archive name 161. Information that identifies the source code is set in the program name 162. In this example, the program name of the source code is set. The revision information of the above source code is set in the update information 163.

FIG. 7 is an example of the above-mentioned change information 170. As shown in the figure, the change information 170 is composed of one or more records having each item of the application archive name 171 and the program name 172, the update information 173, and the difference information 174.

Of the above items, the application archive name 171 and the program name 172, and the update information 173 are the same as the application archive name 161, the program name 162, and the update information 163 of the current deployment information 160, respectively, and thus the description thereof will be omitted. The difference information 174 is set with information indicating the difference between the current deployment information 160 and the information regarding the new application archive generated by the build executed after the source code is changed. In the case of this example, "Yes" is set if the source code has been changed (component update information has been changed), and blank is set if the source code has not been changed.

FIG. 8 is a sequence diagram illustrating a process performed in the data analysis system 1 (a flow of the data analysis process performed after the source code changed by the user is committed). Hereinafter, they will be described in order with the same figure.

First, the user terminal 110 transmits the source code to be committed and the commit instruction of the source code to the source control device 101. When the source control device 101 receives the commit instruction of the source code, the source control device 101 commits (registers) the received source code together with the commit instruction to the source repository 106 (S801).

Subsequently, the user terminal 110 transmits a build execution instruction to the source control device 101 (S802). Upon receiving the execution instruction, the build device 102 transmits a source code acquisition request to the source control device 101 (S803).

Upon receiving the acquisition request, the source control device 101 transmits the source code required for the build to the build device 102 (S804).

When the build device 102 receives the source code from the source control device 101, the build device 102 executes a build using the received source code and generates an application archive (S805). The generated application archive is a material to be deployed in the deployment process described later.

Subsequently, the user terminal 110 transmits a change information generation instruction to the change management device 103 (S806). Upon receiving the generation instruction, the change management device 103 transmits a source information acquisition request to the source management device 101 (S807). Upon receiving the acquisition request, the source management device 101 transmits the source information to the change management device 103 (S808).

Subsequently, the change management device 103 transmits a build information acquisition request at the time of build to the build device 102 (S809). Upon receiving the acquisition request, the build device 102 transmits the build information to the change management device 103 (S810).

Based on the acquired source information and build information, the change management device 103 can acquire information such as revision information of the source code and which application archive the source code is included in.

Subsequently, the change management device 103 generates change information 170 based on the current deployment information 160 and the acquired source information and build information (S811, S812). When the change information 170 is generated, the change management device 103 transmits the generation completion notification of the change information 170 to the user terminal 110 (S813).

Subsequently, a request for acquiring the material to be deployed is transmitted from the user terminal 110 to the build device 102 (S814). Upon receiving the acquisition request, the build device 102 transmits the deployment material to the user terminal 110 (S815). When the user terminal 110 receives the deployment target material, the user terminal 110 deploys the received deployment target material to the execution device 104 (S816).

Subsequently, the user terminal 110 transmits an update instruction for the current deployment information 160 to the change management device 103 (S817). Upon receiving the update instruction, the change management device 103 updates the current deployment information 160 to the latest content based on the content of the change information 170 (S818). When the current deployment information 160 is updated, the change management device 103 transmits an update completion notification of the current deployment information 160 to the user terminal 110 (S819).

Subsequently, the user terminal 110 transmits an execution instruction for data analysis processing to the execution device 104 (S820). At the time of the execution instruction, the user executes the processing elements described in the analysis processing definition information 306 in order from the beginning to the end of the user terminal 110 (hereinafter, referred to as the first execution mode). And, among the processing elements described in the analysis processing definition information 306, the execution of the processing element before the processing element whose source code (component) has been changed is omitted (skipped), and the processing whose source code has been changed. It is possible to specify one of the execution modes (hereinafter, referred to as the second execution mode) in which the processing elements after the element are re-executed in order. When the user selects the second execution mode, the execution device 104 reuses the data generated in the data analysis process performed in the past and stored in the data lake 108, and the source code is changed. Execute the following processing elements.

When the execution device 104 receives the execution instruction from the user terminal 110, the execution device 104 transmits a response to the effect that the execution instruction has been accepted to the user terminal 110 (S821). The data analysis process by the execution device 104 may be performed independently (asynchronously) from the user terminal 110, for example.

Prior to the execution of the data analysis process, the execution device 104 transmits the acquisition request of the change information 170 to the change management device 103 (S822). Upon receiving the acquisition request, the change management device 103 transmits the change information 170 to the execution device 104 (S823).

Upon receiving the change information 170, the execution device 104 executes the data analysis process in the execution mode selected by the user (S824). The details of the processing of S824 will be described later.

After executing the data analysis process, the execution device 104 stores the result (result data) of the data analysis process in the data lake 108 (S825).

FIG. 9 is a flowchart illustrating details of the process performed by the execution device 104 in S824 of FIG. Hereinafter, it will be described together with the figure.

First, the execution device 104 assigns an identifier (hereinafter, referred to as an execution ID) to the data analysis process to be executed in response to the execution instruction received from the user terminal 110 (S901).

Subsequently, the execution device 104 acquires the change information 170 from the change management device 103 (S902).

Subsequently, the execution device 104 determines which execution mode is selected (S903). When the first execution mode is selected (S903: first execution mode), the execution device 104 executes the processing elements described in the analysis processing definition information 306 in order from the beginning to the end (S904 to S907). On the other hand, when the second execution mode is selected (S903: second execution mode), the process proceeds to S910.

In S910, the execution device 104 selects the processing element first described in the analysis processing definition information 306. Subsequently, the execution device 104 determines whether or not the selected processing element has been changed (whether or not the component of the application archive that realizes the processing element has been changed) based on the change information 170 (S911). .. If the selected processing element has been changed (S911: YES), the processing proceeds to S905, and the execution device 104 sequentially processes the selected processing element to the last processing element described in the analysis processing definition information 306. The process is executed (S905 to S907).

On the other hand, if the selected processing element has not been changed (S911: NO), the processing proceeds to S912. In S912, the execution device 104 omits (skips) the execution of the selected processing element. Subsequently, the execution device 104 determines whether or not the subsequent processing element is described in the analysis processing definition information 306 (S913). When a subsequent processing element is described (S911: YES), the execution device 104 reselects the next processing element (S914), and processes the newly selected processing element from S911. On the other hand, when the subsequent processing elements are not described (that is, when all the processing elements of the data processing elements are not changed) (S9111: NO), the processing ends.

After omitting (skipping) the execution of the processing element (S912), the process proceeds from S911 to S905 (S911: YES), and the data analysis processing is performed from the processing element in the middle of the processing elements described in the analysis processing definition information 306. When re-executing, the execution device 104 puts the processing element to be re-executed before the processing element that starts re-execution in the past data analysis processing (data analysis processing performed before the source code is changed). For example, the data (input data or intermediate) generated by the processing element (immediately before) (processing element described in the analysis processing definition information 306 so as to be executed before (for example, immediately before) the processing element that starts re-execution). Data) is given as input. In the re-execution, for example, any data among the data generated in the past by the processing element described in the analysis processing definition information 306 so as to be executed before the processing element that starts the re-execution is used. It may be possible to set (automatic setting, user setting, etc.) according to the mode of the data analysis processing.

FIG. 10 is an example of a data management mode in the data lake 108. As shown in the figure, in this example, in the data lake 108, for each execution ID assigned to the analysis process in S901 of FIG. 9 (for each execution instruction of the data analysis process received by the execution device 104 from the user terminal 110). Data space is secured. The data space is automatically secured by the data input / output unit 305 or the data lake 108 when the execution device 104 assigns the execution ID, for example. In this example, the data lake 108 is provided with a data space 1010 reserved for the data analysis process having the execution ID "1" and a data space 1020 reserved for the data analysis process having the execution ID "2". There is. Each data space stores data (input data, intermediate data, result data, etc.) generated by each processing element of the data analysis process (in this example, a combination of a "key" and a value "value" is included. Data composed of one or more records is shown as an example.)

FIG. 11 is a diagram showing the relationship between the flow of the data analysis process and the data (table) stored in the data lake 108 when the data analysis process is executed by the execution device 104. Hereinafter, it will be described together with the figure.

First, the execution device 104 assigns "1" as an execution ID to the data analysis process, and secures a data space 1150 having an execution ID of "1" in the data lake 108 (S901 in FIG. 9).

Subsequently, when the execution device 104 executes the processing element 1 (S1111), the table 1 (input data) 1151 is stored in the data space 1150 whose execution ID of the data lake 108 is “1” (S904 in FIG. 9). ).

Subsequently, the execution device 104 executes the processing element 2 (S1112), the processing element 3 (S1113), and the processing element 4 (S1114) in this order, so that the table 2 (table 2 ( Intermediate data) 1152, table 3 (intermediate data) 1153, and table 4 (intermediate data) 1154 are generated in this order.

Subsequently, when the execution device 104 executes the processing element 5 (S1115), the table 5 (result data) 1155 is generated in the data space 1150 having the execution ID “1” (S905 to S907 in FIG. 9).

In FIG. 12, after the data analysis process shown in FIG. 11 is executed, the processing element 3 (S1113) is changed (the source code of the processing element 3 is changed) to perform the build, and then the data analysis process is re-executed. It is a figure which shows the relationship between the flow of the data analysis processing at the time of execution, and the data (table) stored in the data lake 108. In this example, the execution device 104 re-executes the data analysis process using the data (table) generated in the data analysis process shown in FIG.

First, the execution device 104 assigns "2" as an execution ID to the data analysis process to be re-executed, and secures a data space 1250 having an execution ID of "2" in the data lake 108 (S901 in FIG. 9). When the data analysis process shown in FIG. 12 is performed, the data space 1150 remains secured in the data lake 108, and all the data stored in the data space 1150 in the data analysis process shown in FIG. 11 is the data lake 108. It is assumed that it is stored (remains) in.

Subsequently, the execution device 104 confirms that the processing element 1 (S1111) and the processing element 2 (S1112) have not been changed based on the change information 170, and the processing element 1 (S1111) and the processing element 2 (S1112). Execution is omitted (skipped) ((S911: NO) and S912 in FIG. 9).

Subsequently, the execution device 104 confirms that the source code of the processing element 3 (S1113) has been changed, and processes the table 2 (intermediate data) 1152 generated by the data analysis process shown in FIG. 11 as an input. Execute element 3 (S1113). As a result, the table 3 (intermediate data) 1161 is stored in the data space 1250 whose execution ID of the data lake 108 is “2”.

Subsequently, the execution device 104 uses the processing element 4 (S1114) and the processing element 5 (S1115).
Is executed, and as a result, table 4 (intermediate data) 1162 and table 5 (result data) 1165 are stored in the data space 1250 whose execution ID is "2" ((S911: YES) in FIG. 9, S905 to S907). ..

FIG. 13 is an example of a user interface (execution request instruction screen 1300) provided by the user terminal 110 to the user when the user requests the execution device 104 to execute the data analysis process via the user terminal 110 in FIG. .. As shown in the figure, the execution request instruction screen 1300 includes a data analysis process designation field 1311, an execution mode selection field 1312, and an execution request instruction button 1313.

The user can specify the data analysis process to be executed by the execution device 104 by designating the data analysis process identifier (data analysis process ID) in the data analysis process designation field 1311. Further, the user can specify the above-mentioned execution mode via the execution mode selection field 1312. Further, the user can make an execution request for the data analysis process to the execution device 104 by operating the execution request instruction button 1313.

As described above, in the data analysis system 1 of the present embodiment, the changed processing element is automatically specified, and the specified processing element is generated by the data analysis processing executed in the past. Since the data is used for re-execution, the data analysis process can be executed efficiently. According to the data analysis system 1 of the present embodiment, for example, even when the data analysis system 1 is shared by a plurality of users in a large-scale project or the like and it is difficult to accurately grasp the update status of the materials to be deployed. , It is possible to accurately identify the optional processing elements, and the data analysis work can be carried out efficiently.

By the way, it goes without saying that the present invention is not limited to the embodiments described above, and various modifications can be made without departing from the gist thereof. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the described configurations. Further, it is possible to add / delete / replace a part of the configuration of the above embodiment with another configuration.

Further, each of the above configurations, functional units, processing units, processing means and the like may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be placed in a memory, a hard disk, a recording device such as an SSD, or a recording medium such as an IC card, an SD card, or a DVD.

Further, in each of the above figures, the control lines and information lines are shown as necessary for explanation, and not all the control lines and information lines in the implementation are necessarily shown. For example, in practice almost all configurations may be considered interconnected.

Further, the arrangement form of various functional units, various processing units, and various databases of the data analysis system 1 described above is only an example. The arrangement form of the various function units, the various processing units, and the various databases can be changed to the optimum arrangement form from the viewpoints of the performance, processing efficiency, communication efficiency, and the like of the hardware and software included in the data analysis system 1.

Further, the above-mentioned database configuration (schema, etc.) can be flexibly changed from the viewpoints of efficient use of resources, improvement of processing efficiency, improvement of access efficiency, improvement of search efficiency, and the like.

1 Data analysis system, 2 Server segment, 3 User segment, 5 Communication network, 10 Information processing device, 101 Source control device, 102 Build device, 103
Change Management Device, 104 Execution Device, 106 Source Repository, 107 Archive Repository, 108 Data Lake, 110 User Terminal, 160 Current Deployment Information, 170 Change Information, 201 Program Management Department, 202 Build Execution Department, 203 Change Management Department, 301 Execution Request reception unit, 302 process execution control unit, 303 command call unit, 304 analysis logic execution unit, 305 data input / output unit, 306 analysis process definition information

Claims (12)

A storage unit that stores analysis process definition information including information that defines the execution order of multiple processing elements that make up the data analysis process.
A process execution unit that executes the process element according to the analysis process definition information by the deployed material and stores data generated by each of the process elements.
A build execution unit that executes a build based on the source code of the processing element and generates materials to be deployed,
The process execution unit stores the currently deployed information, which is the update information of the currently deployed material, accepts the change of the source code, executes the build based on the changed source code, and generates the new material. Then, the change management unit that generates change information, which is information indicating the difference between the current deployment information and the update information of the new material,
Including
When executing the data analysis process, the process execution unit determines whether or not the process element of the data analysis process has been changed based on the change information, and if it determines that the data analysis process has been changed, the processing element has been changed. The execution of the processing element defined in the analysis processing definition information so as to be executed before is omitted, and the execution of the processing element is defined in the analysis processing definition information so as to be executed before the changed processing element. Using the data generated in the past by the processing element, the processing is executed from the modified processing element.
Data analysis system. The data analysis system according to claim 1.
The material is an application archive
The current deployment information includes update information of the components contained in the application archive.
The change information includes information indicating the difference between the update information of the current deployment information and the update information of the component included in the application archive of the new material.
Data analysis system. The data analysis system according to claim 2.
Each of the processing elements is realized by one application archive.
The build execution unit manages information indicating which application archive contains the component based on the source code.
The change management unit generates the change information including information indicating the correspondence between the application archive and the component based on the information acquired from the build execution unit.
Data analysis system. The data analysis system according to any one of claims 1 to 3.
The change management unit has a source code change information receiving unit that accepts changes to the source code.
Data analysis system. The data analysis system according to any one of claims 1 to 3.
The processing execution unit has an analysis processing definition information receiving unit that receives and stores the analysis processing definition information.
Data analysis system. The data analysis system according to any one of claims 1 to 3.
It is provided with a user interface that allows the user to select whether or not to perform the above omission.
When it is selected that the omission is not performed via the user interface, the process execution unit starts the process element defined in the analysis process definition information from the beginning regardless of whether or not the process element is changed. Execute in order,
Data analysis system. Information processing device
A step to store analysis process definition information, including information that defines the execution order of multiple processing elements that make up a data analysis process.
A step of executing the processing element according to the analysis processing definition information by the deployed material and storing the data generated by each of the processing elements.
A step of executing a build based on the source code of the processing element to generate materials to be deployed,
The currently deployed information, which is the update information of the currently deployed material, is stored, the change of the source code is accepted, the build is executed based on the changed source code to generate the new material, and the currently deployed material is generated. A step of generating change information, which is information indicating the difference between the information and the update information of the new material.
When executing the data analysis process, it is determined whether or not the processing element of the data analysis process has been changed based on the change information, and if it is determined that the processing element has been changed, the data analysis process is executed before the changed processing element. The processing element defined in the analysis processing definition information is omitted in the past so that the processing element defined in the analysis processing definition information is executed before the changed processing element. A step of executing processing from the modified processing element using the generated data,
How to analyze data. The data analysis method according to claim 7.
The material is an application archive
The current deployment information includes update information of the components contained in the application archive.
The change information includes information indicating the difference between the update information of the current deployment information and the update information of the component included in the application archive of the new material.
Data analysis method. The data analysis method according to claim 8.
Each of the processing elements is realized by one application archive.
The information processing device
A step of managing information indicating which of the application archives contains components based on the source code.
A step of generating the change information, which includes information indicating the correspondence between the application archive and the component, based on the information.
A data analysis method that further performs. The data analysis method according to any one of claims 7 to 9.
A step in which the information processing device accepts changes in the source code.
A data analysis method that further performs. The data analysis method according to any one of claims 7 to 9.
A step in which the information processing apparatus receives and stores the analysis processing definition information.
A data analysis method that further performs. The data analysis method according to any one of claims 7 to 9.
The information processing device includes a user interface that allows the user to select whether or not to perform the omission.
When it is selected that the information processing apparatus does not perform the omission via the user interface, the processing element defined in the analysis processing definition information is used from the beginning regardless of whether or not the processing element is changed. Steps to perform in sequence,
A data analysis method that further performs.

Images