JP5247192B2 - Cyclic update data management system - Google Patents

Description

  The present invention relates to a periodic update data management system including a storage unit and an update processing unit.

  Generally, when updating a specific one of a large number of information (records) stored in a computer, it is necessary to first search for a record based on a key (ID) that specifies an update target. At this time, if the amount of information to be handled increases, it becomes necessary to access a secondary storage device that is extremely slow compared to the main storage device.

  Therefore, in a database that manages a large number of records, a tree structure index such as “B-tree” is generally used to retrieve a record corresponding to a key with a small number of secondary storage accesses. In addition, the buffering that keeps the record once accessed in the main memory makes it possible to avoid the access to the secondary storage device for the frequently accessed record.

Secondary storage devices, as represented by magnetic disk devices, access to discontinuous addresses (random access) for access (sequential access) speeds (several tens of MB / s to several hundreds MB / s) for consecutive addresses. ) The speed (several MB / s) is extremely low.
As a result, when a large number of records are updated, the obtained performance varies greatly depending on whether the access pattern to the secondary storage is sequential or random.

  In the “B-tree index” for search based on the key (ID), performance similar to sequential access is obtained in access in key order, but it is enormous for access in arbitrary order that does not match the key order. As long as there is no main memory buffer, only random access performance can be obtained.

  Therefore, in consideration of the above characteristics, a method has been proposed in which access requests for a plurality of records are collected and rearranged in the order of keys, and then actual access is performed to prevent performance degradation (see, for example, Patent Document 1). .

  Note that when a new record is added instead of being updated, it is not necessary to search for a record. Therefore, sequential access can be realized by storing records in the order of writing. This is a storage method according to the time of writing, and is a general method known as a log format. However, if an attempt is made to update the added record again, random access occurs, which is not suitable for the update.

Japanese Patent Laid-Open No. 10-31602, page 1, FIG.

  In the conventional periodic update data management system, as described in Patent Document 1, records are stored in the order of keys, and access requests for a plurality of records are actually accessed after being rearranged in the order of keys. In such a case, there is a problem that the access target is sparse, it is rare to access continuous records, and the effect of preventing performance degradation cannot be obtained.

  The present invention has been made to solve the above-described problems. When the update time for each record is known in advance (when the system itself schedules the update), the storage unit (secondary storage) It is an object of the present invention to provide a periodic update data management system capable of speeding up the update processing of a large number of records so that reading and writing to the apparatus) are sequential access.

  The periodic update data management system according to the present invention is a periodic update data management system including a storage unit and an update processing unit, and the storage unit stores a plurality of records corresponding to a plurality of external information sources on a one-to-one basis. The update processing unit selects a region corresponding to the current time from among the plurality of regions as a processing target region, and records the target record in the processing target region. , Refer to the target information source corresponding to the target record from a plurality of information sources, determine the updated value of the target record based on the acquired information acquired from the target information source, and at the time of the previous reference of the acquired information The next update time of the target record is determined based on the presence or absence or magnitude of the change from, and the updated value of the target record is stored at the end of the area corresponding to the determined next update time After it has finished processing all of the target record of the processing target area, it is intended to remove the processing target area.

  According to the periodic update data management system according to the present invention, the update processing of a large number of records can be speeded up.

Embodiment 1 FIG.
1 is a block diagram showing a periodic update data management system according to Embodiment 1 of the present invention.
1, the periodic update data management system includes a storage unit 1, a read buffer 4 connected to the storage unit 1, write buffers 5a, 5b connected to the storage unit 1, and a write buffer. Are provided with an update processing unit 6 for controlling 5a, 5b,..., A second storage unit 8 different from the storage unit 1, and a request receiving unit 10 connected to the second storage unit 8.
Hereinafter, the description will be made by paying attention to two write buffers 5a and 5b among the plurality of write buffers.

  In addition, as an external configuration of the periodic update data management system, a plurality of information sources (A, B, C,...) 7a, 7b, 7c,. The following description will be made with a focus on three information sources 7a to 7c among a plurality of information sources.

The storage unit 1 has a plurality of storage areas (hereinafter referred to as “buckets”) 2a, 2b, 2c,..., And each bucket 2a, 2b, 2c,. 3 is stored.
Hereinafter, the description will be made by paying attention to three buckets 2a to 2c out of a plurality of buckets.

The update processing unit 6 transmits a registration request to the request receiving unit 10 based on the plurality of information sources 7a to 7c, and controls data held in the write buffers 5a and 5b (records).
The second storage unit 8 includes an information source registration table 9a, and stores identifiers A to C corresponding to the information sources 7a to 7c, for example.
The request receiving unit 10 controls the data held in the write buffers 5a and 5b in response to registration requests from the update processing unit 6 and the control terminal 11.

The information sources 7a to 7c and the control terminal 11 existing outside the periodic update data management system are connected to the periodic update data management system via a communication line such as the Internet or a LAN.
Here, as a representative example, the information sources 7a to 7c are configured by Web pages on a Web server, and the control terminal 11 is configured by a general computer.
For example, the information sources 7a to 7c are logical entities, and a URL for identifying each of the information sources 7a to 7c is specified and a Web page acquisition request is sent to a Web server that is a physical entity. Correspond.

The purpose of the system according to the first embodiment of the present invention shown in FIG. 1 is to manage information for determining the timing for acquiring each Web page as a record in order to capture temporal changes of each Web page. It is in.
Here, the record is updated each time a Web page is acquired, and the next acquisition timing is adaptively controlled. The content of the actually acquired Web page itself may be stored in another storage unit other than the storage unit 1 in a log format or the like.

The periodic update data management system of FIG. 1 can be realized using a general computer.
That is, the storage unit 1 and the second storage unit 8 are realized by a magnetic disk device, the read buffer 4 and the write buffers 5a to 5b are realized by a random access memory, and the update processing unit and the request reception unit 10 are randomly accessed. It can be realized as a computer program stored in a memory and executed by a processor.

The buckets 2a to 2c occupy different areas of the storage unit 1, and can be newly created, expanded and deleted independently.
The buckets 2a to 2c in the storage unit 1 can be realized as files provided by a general operating system such as Linux or Windows (registered trademark).

A plurality of records 3 are stored in each of the buckets 2a to 2c.
Each record 3 has a one-to-one correspondence with each information source 7a to 7c, and is updated based on information of the corresponding information source 7a to 7c in an independent cycle.

The record 3 includes the next update time expressed in a predetermined unit (time resolution), for example, “day”, and the records 3 stored in the buckets 2a to 2c all have the same next update time. .
That is, the buckets 2a to 2c to be stored are determined according to the next update time of the record 3. In addition, the arrangement of the records 3 in each of the buckets 2a to 2c is in the order of storage, and in order to search for the record 3 having a specific information source identifier, it is necessary to scan the entire buckets 2a to 2c.

The write buffers 5a to 5b exist at every next update time, similarly to the buckets 2a to 2c.
The information source registration table 9a stores the identifiers of known information sources. For example, a specific information source identifier can be efficiently searched by using a known tree structure such as “B-tree”. To do.

  Access to the information source registration table 9a is essentially random access, but is small compared with the entire record 3, so the entire storage is cached in a random access memory, so that the storage unit 1 and the second storage Access to the unit 8 (magnetic disk device) can be avoided.

FIG. 2 is an explanatory diagram showing in detail the information included in the record 3 in the bucket 2.
In FIG. 2, record 3 includes information source identifier 31, next update time 32, final state 33, final update time 34, and update cycle 35 as information.

The information source identifier 31 represents the URL of a Web page that is an information source.
The next update time 32 is a time when the record is scheduled to be updated next. As described above, each record 3 in the same bucket has the same next update time expressed in a predetermined unit (for example, “day”). Have.

The final state 33 represents the information acquired last from the information source corresponding to the information source identifier 31.
Note that in order to determine the next update time 32 of the record 3, if only the presence / absence of the value is important, not the value of the information itself, the final state 33 has a hash value instead of the value itself. An expression obtained by compressing the data length by applying a function (hash function) can be used.

  As described above, if the information value itself or the value history is necessary for purposes other than the determination of the next update time 32 (for example, generation of a full-text search index for the acquired Web page), These are stored in the storage means.

The last update time 34 and the update cycle 35 in the record 3 are data that is referred to and updated when the next update time 32 is determined.
Note that the record 3 may further include auxiliary data (for example, Web site authentication information and cookie information) necessary for obtaining information from the information sources 7a to 7c.

Next, operations of the update processing unit 6 and the request receiving unit 10 will be described with reference to the flowcharts of FIGS. 3 and 4. First, the operation of the update processing unit 6 shown in FIG. 3 will be described.
In FIG. 3, the update processing unit 6 first selects one of the buckets 2 a to 2 c corresponding to the minimum (t) next update time 32 as a processing target bucket (processing target region) t (hereinafter simply referred to as “bucket t”). (Also referred to as “)” (step S1).

  At this time, as the bucket t corresponding to the minimum value t of the next update time 32, a bucket corresponding to the current time can be simply selected, but the processing is performed within the unit period (for example, one day) of the next update time 32. If there is a bucket that does not complete, there may be a bucket that is not processed.

  Then, it waits until the current time reaches “t” (step S2). Step S2 is executed because the bucket 2 may be sparse when there is no record 3 to be updated within the unit period.

Thereafter, the update processing unit 6 reads the contents of the bucket t all at once into the reading buffer 4 (step S3). Thereby, the reading with respect to the memory | storage part 1 is set as sequential access.
Subsequently, it is determined whether or not all the records in the read buffer 4 have been processed (step S4). If it is determined that all the records have been processed (that is, YES), the process proceeds to step S10 and unprocessed. If it is determined that the record remains (that is, NO), the process proceeds to step S5.

In step S5, the update processing unit 6 selects one of the unprocessed records, and acquires information from the information source based on the information source identifier 31 of the selected record.
As described above, when the acquired information needs to be stored in another storage unit, it is stored at the time of step S5.

  Here, since the acquired information is a Web page and includes the URL of another Web page as a hyperlink, the update processing unit 6 extracts these URLs and uses each URL as an information source identifier. By sending the registration request to be sent to the request receiving unit 10, a new information source can be found.

In step S5, the update processing unit 6 determines the next update time t ′ based on the acquired information.
The next update time t ′ is set based on whether or not the update of information is detected by comparing the final state 33 stored in the record 3 with a new final state based on the acquired information, for example. That is, when update of information is detected, 1/2 of the update cycle 35 is added to the current time, and when update of information is not detected, a value obtained by adding “1” to the update cycle 35 Can be determined by a method such as adding to the current time.

  Further, in step S5, the update processing unit 6 sets the next update time 32 to “t ′”, sets the final state 33 to a new value, sets the final update time 34 to the current time, and updates the update cycle 35. Is set to “½ of the original value” when the information update is detected, and is set to “the value obtained by adding 1 to the original value” when the information update is not detected. .

  Note that a more advanced update cycle determination method is known in the literature (for example, Database Society of Japan Letters, Vol. 6, No. 1, pp. 173-176 “Design and Implementation of an Update Crawler for Large-Scale Web Archives”). Takayuki Tamura, Yu Kitakawa, or Japanese Patent Application No. 2007-018012, “Web Page Recollection Method”).

Subsequently, it is determined whether or not the record is a rejection target (step S6).
In general, information acquisition from an information source may end abnormally, and if the information is not updated, the update cycle 35 becomes very large. Therefore, for example, by setting a predetermined condition such as when the abnormal termination continues “three times” or when the update cycle 35 becomes “two years or more”, the “rejected object” is set when the predetermined condition is satisfied. Can be determined.

  If it is determined in step S6 that the record is to be rejected (that is, YES), the process immediately returns to step S4. If it is determined that the record is not to be rejected (that is, NO), the process proceeds to the next step S7. move on.

In step S7, the free capacity of the write buffer t ′ corresponding to the next update time t ′ is checked to determine whether the write buffer t ′ is full (no free space).
If it is determined in step S7 that there is no space (ie, YES), the process proceeds to the next step S8, and if it is determined that there is a space (ie, NO), the process skips step S8 and proceeds to step S9.

In step S8, the write buffer t ′ is flushed. That is, all records in the write buffer t ′ are added to the end of the bucket t ′, and the write buffer t ′ is emptied. By the operation in step S8, writing to the storage unit 1 is set as sequential access.
Subsequently, the value of the record updated in step S5 is added to the end of the write buffer t ′ (step S9), and the process returns to step S4.

  On the other hand, if it is determined in step S4 that the record processing by the read buffer 4 is completed (that is, YES), step S10 is executed. In step S10, the write buffer t + 1 is flushed and then the write is performed. Delete buffer t + 1.

Step S10 is processing for the bucket t + 1 to be processed next.
Finally, the reading buffer is emptied (step S11), the processed bucket t is deleted (step S12), and the process returns to step S1.

Next, the operation of the request receiving unit 10 shown in FIG. 4 will be described.
In FIG. 4, the request receiving unit 10 first receives a registration request (step S21).
At this time, the registration request includes the information source identifier R of the information source to be registered.

  Subsequently, the request receiving unit 10 refers to the information source identifier R of the registration request, searches the information source registration table 9a in the second storage unit 8, and the information source identifier R of the registration request is the information source registration table. It is determined whether it is present (found) in 9a (step S22).

If it is determined in step S22 that the information source identifier R exists in the information source registration table 9a (that is, YES), the information source identifier R has already been registered in the information source registration table 9a, and the process returns to step S21.
On the other hand, if it is determined in step S22 that the information source identifier R does not exist in the information source registration table 9a (that is, NO), the process proceeds to the next step S23, and the information source identifier R is added to the information source registration table 9a. sign up.

Subsequently, the record corresponding to the information source identifier R is initialized, and the next (first time) update time t ′ is determined (step S24).
The update time t ′ can be determined by adding a predetermined constant (for example, “1”) or a random number within a predetermined range to the current time t.
Finally, the request receiving unit 10 executes the record writing process (steps S25 to S27) similar to steps S7 to S9 of the update processing unit 6 (FIG. 3), and returns to step S21.

  As described above, according to the first embodiment (FIGS. 1 to 4) of the present invention, the periodic update data management system includes the storage unit 1 and the update processing unit 6, and the storage unit 1 is externally connected. The plurality of records 3 corresponding one-to-one with the plurality of information sources 7a to 7c are stored in different buckets 2a to 2c based on the respective next update times.

  The update processing unit 6 selects the bucket corresponding to the current time among the plurality of buckets 2a to 2c as the processing target bucket, and the target record for the target record in the processing target bucket from the plurality of information sources 7a to 7c. Based on the acquisition information acquired from the target information source, the updated value of the target record is determined, and based on the presence / absence or magnitude of the acquisition information from the previous reference The next update time t ′ of the target record is determined, the updated value of the target record is stored at the end of the bucket corresponding to the determined next update time t ′, and all target records in the processing target bucket are stored. After processing is completed, the target bucket is deleted.

In this way, by having a plurality of buckets 2a to 2c that store a plurality of records 3 based on the next update time t ′, it is possible to simultaneously read records that are updated simultaneously in units of buckets. The update process can be speeded up.
Further, random access in units of records can be avoided, and performance degradation of the storage unit 1 (magnetic disk device) can be prevented.

Further, the periodic update data management system according to Embodiment 1 of the present invention has a one-to-one correspondence with a single read buffer 4 connected to the storage unit 1 and a plurality of buckets 2a to 2c in the storage unit 1. And a plurality of write buffers 5a and 5b having a predetermined capacity.
In this case, the update processing unit 6 collectively reads all target records stored in the processing target bucket into the read buffer 4, and for each target record in the read buffer 4, from among the plurality of information sources 7a to 7c. The updated value and the next update time t ′ are determined with reference to the target information source corresponding to the target record.

  Further, the update processing unit 6 selects a first write buffer corresponding to the determined next update time t ′ from the plurality of write buffers 5a and 5b, and the first write buffer already has a predetermined capacity. If records are stored, write all records in the first write buffer to the end of the bucket corresponding to the first write buffer, empty the first write buffer, The updated value of the target record is stored at the end of the first write buffer, and after processing all the records in the read buffer 4, the second corresponding to the next processing target bucket to be processed next is stored. All the next target records in the write buffer are written at the end of the next processing target bucket, the second write buffer is emptied, and then the second write buffer and the processing target bucket are deleted.

In this way, by providing the write buffers 5a to 5b for collectively writing a plurality of records 3 having the same next update time t ′ without writing the updated records back to the original positions, Random access can be avoided, and even if the update cycle is different for each record, it can be ensured that the next read operation will be sequential access.
Moreover, since the buckets read in a lump can be reused immediately, fragmentation of the bucket (storage area) can be avoided.

Moreover, the periodic update data management system according to Embodiment 1 of the present invention further includes a second storage unit 8 different from the storage unit 1 and a request reception unit 10 connected to the second storage unit 8. I have.
The second storage unit 8 includes an information source registration table 9 a that stores identifiers of information sources corresponding to all records stored in the storage unit 1.
The request receiving unit 10 receives a registration request including an identifier corresponding to any one of a plurality of information sources, and if the identifier is not stored in the information source registration table 9a, the request is received from the information source registration table. In addition to 9a, the record corresponding to the identifier is initialized to a predetermined value and written to one of the plurality of buckets 2a to 2c in the storage unit 1 or one end of the write buffers 5a and 5b. .

As described above, by providing the information source registration table 9a, it is possible to efficiently perform a process of dynamically adding a new information source without searching for the entire bucket while avoiding duplication.
In particular, when the information of the information source is a web page on the website, the identifier of the information source is the URL of the web page, and the update processing unit 6 selects the web page corresponding to the target record in the processing target bucket. Download the contents, determine the updated value of the target record, extract a link from the Web page corresponding to the target record, and send each of the obtained URLs to the request receiving unit 10 as a registration request Thus, a new information source candidate can be found from the hyperlink, and autonomous Web information collection becomes possible.

  In addition, the update processing unit 6 has a record rejection process (step S6), and when the updated value of the record satisfies a predetermined condition, the update processing unit 6 determines whether the record is a plurality of buckets 2a to 2c or a write buffer. In this case, it is possible to avoid reading / writing a record that no longer needs to be updated.

In addition, although the update process part 6 implement | achieved the rejection process of a record by not writing a record in step S6, furthermore, the information source identifier (identifier corresponding to the record) corresponding from the information source registration table 9a is obtained. It may be deleted.
This makes it possible to register again an information source that has been determined to be unnecessary.

In addition, the update processing unit 6 according to the first embodiment of the present invention sets the second write buffer corresponding to the next processing target bucket as a new read buffer after processing all the records in the read buffer 4, Delete the original read buffer and target bucket. That is, in steps S10 and S11, the write buffer t + 1 is deleted and the read buffer 4 is emptied. Instead, the read buffer 4 may be deleted and the write buffer t + 1 may be a new read buffer.
As a result, it is possible to avoid the overhead of writing the record in the write buffer t + 1 into the bucket t + 1 and reading it back into the read buffer.

In the above description, the information source is Web page information, but it may be a file on a file server.
In this case, the identifier of the information source is the path name of the file, and the update processing unit 6 downloads the contents of the file corresponding to the target record in the processing target bucket and determines the updated value of the target record. At the same time, if the file corresponding to the target record is a directory, the path names of the files and subdirectories in the directory are sent to the request receiving unit 10 as registration requests.

In this way, the directory (folder) information is used instead of the Web page hyperlink, and the update processing unit 6 requests a registration request using the path name of the file in the directory as the information source identifier in step S5. By transmitting to the reception unit 10, it is possible to autonomously respond to the addition of a file.
Further, the deletion of the file can be dealt with by the rejection process in step S6.

Embodiment 2. FIG.
In the first embodiment (FIG. 3), when the determination result in step S4 is NO (record processing is not completed), the next update time t ′ determination process (step S5) is executed unconditionally. As shown in FIG. 5, the next update time t ′ determination process (steps S <b> 52 and S <b> 53) may be executed according to whether a predetermined number of processes or a predetermined time elapsed determination condition (step S <b> 51) is correct.

FIG. 5 is a flowchart showing the operation of the update processing unit 6 according to the second embodiment of the present invention, and illustration of processes similar to those described above (see FIG. 3) is omitted.
The overall configuration of the periodic update data management system according to the second embodiment of the present invention is the same as that described above (see FIG. 1), and only the partial operation of the update processing unit 6 is different.

  In FIG. 5, when the determination result in step S4 is NO (the unprocessed record remains in the read buffer 4), the update processing unit 6 processes the predetermined number of records after starting the process of the read buffer 4. It is determined whether or not a predetermined time has passed (step S51).

If it is determined in step S51 that the condition is not satisfied (that is, NO), one of the unprocessed records is selected, and the same processing (step S52) as in step S5 described above is executed.
However, the record selected in step S53 is assumed to have the largest value K below among unprocessed records.

  K = (current time t−last update time 34) / update cycle 35

  This value K represents the magnitude of the accumulated waiting time relative to the record update period, and corresponds to processing preferentially from the record that is greatly affected by the accumulated waiting time.

On the other hand, if it is determined in step S51 that the condition is satisfied (that is, YES), it means that the time (load) required to update the read buffer 4 is too large and adversely affects the update of the subsequent bucket. Instead of executing the process of selecting one of the unprocessed records and acquiring information from the information source (step S52), only the next update time t ′ is set (step S53), and the process proceeds to step S7.
At this time, the next update time t ′ can be determined by adding a predetermined constant (for example, “1”) or a predetermined range of random numbers to the current time t.

  As described above, according to the second embodiment (FIG. 5) of the present invention, the update processing unit 6 has processed a predetermined number of records in the processing target bucket, or a predetermined time from the start of processing of the processing target bucket. When the time has elapsed (when the determination result of step S51 is NO), the next update time t ′ is updated to a predetermined value without referring to the information source corresponding to the unprocessed record (step S53), Write to the end of one of the buckets 2a-2c or one of the write buffers 5a, 5b.

As described above, by having the processing (steps S51 and S53) for aborting the update processing of the bucket, it is possible to avoid adversely affecting the update time of the subsequent bucket due to the concentration of the update load on the specific bucket.
In addition, when the condition of step S51 is not satisfied, the update processing unit 6 processes the records in the processing target bucket in order of priority based on the relative cumulative waiting time (step S52). Even if there is, update of the record can be executed fairly.

Embodiment 3 FIG.
Although not particularly mentioned in the first and second embodiments, as shown in FIG. 6, in order to temporarily store records to be written in all buckets to be processed one after another, the first document A second write buffer 5y may be provided separately from the embedded buffer 5x.
FIG. 6 is a block diagram showing a periodic update data management system according to Embodiment 3 of the present invention. Components similar to those described above (see FIG. 1) are denoted by the same reference numerals as those described above, or A detailed description will be omitted with “A” attached later.

In FIG. 6, the first write buffer 5x and the second write buffer 5y hold stored data for the buckets 2a to 2c in the storage unit 1A under the control of the update processing unit 6A and the request receiving unit 10A. .
In this case, unlike the number of write buffers 5a and 5b described above, the number of write buffers is two (first and second write buffers 5x and 5y) regardless of the number of buckets 2a to 2c. It is.

The first write buffer 5x corresponds to a single bucket (the bucket next to the processing target bucket) in the storage unit 1A, and the first write buffer 5x stores a record to be written to the single bucket. The
The second write buffer 5y stores a record to be written in a bucket other than a single bucket.

The second storage unit 8A includes rejection buckets 12a, 12b,... In addition to the information source registration table 9b corresponding to the information source registration table 9a described above.
Rejection buckets 12a, 12b,... Store a plurality of records corresponding to buckets 2a to 2c, as described above.
Further, not only the above-described registration request but also a search request and an update request are transmitted from the control terminal 11 to the request receiving unit 10A.

FIG. 7 is an explanatory diagram showing the information source registration table 9b in the second storage unit 8A in detail.
In FIG. 7, the information source registration table 9b is different from the information source registration table 9a described above (see FIG. 1), in addition to the information source identifier 91, information sources (A, B, C,...) 7a, The next update time 92 is stored for each of 7b, 7c,.
The next update time 92 is a value of the next update time 32 (see FIG. 2) of the record 3 corresponding to the information source identifier 91, and some information sources may indicate that it has been rejected.

FIG. 8 is an explanatory diagram showing in detail the information contained in the buckets 2a to 2c in the storage unit 1A. The same components as those described above (see FIG. 2) are denoted by the same reference numerals as those described above, and detailed description thereof is omitted. To do.
In FIG. 8, the bucket 2 may include an update record 30 having a different format in addition to the record 3, unlike the above (FIG. 2).

The update record 30 includes a fixed update period 36 and an update content 37 for instructing an update to a record corresponding to the same information source, in addition to the information source identifier 31 and the next update time 32 as described above.
The update content 37 includes the update target item name of the record 3 and its value.

Next, operations of the update processing unit 6A and the request receiving unit 10A according to the third embodiment of the present invention will be described with reference to the flowcharts of FIGS.
First, the operation of the update processing unit 6A will be described with reference to FIG.
In FIG. 9, steps S101 to S104, S106, S107 and S118 are the same processes as steps S1 to S6 and S12 described above (see FIG. 3), respectively.

  First, the update processing unit 6A reads the contents of the bucket t into the reading buffer 4 (steps S101 to S103). If it is determined in step S104 that an unprocessed record remains (that is, NO), the update processing unit 6A If an update record having the same information source identifier 31 (see FIG. 8) as the processing target record exists in the read buffer 4, the update contents of the update record are selected. The processing target record is updated according to 37, and the updated content 37 is reflected in the updated record (step S105).

Subsequently, information is acquired from the information source, a record update process and a next update time t ′ determination process (step S106) are performed, and it is determined whether the record is a rejection target (step S107).
If it is determined in step S107 that the record is to be rejected (that is, YES), the process proceeds to step S108, and if it is determined that the record is not to be rejected (that is, NO), the process proceeds to step S109.

In step S108, the updated record is added to the end of one of the rejection buckets 12a, 12b,. In addition, although the rejection bucket may be single, when it reaches a predetermined size, it may be divided into a plurality by writing in a new rejection bucket.
Furthermore, in step S108, information for identifying the written rejection bucket is set for the next update time 92 of the information source registration table 9b corresponding to the information source identifier of the record, and the process returns to step S104.

On the other hand, in step S109, the next update time t ′ is compared with the time obtained by adding “1” to the current time t to determine whether or not “t ′ = t + 1”.
If it is determined in step S109 that t ′ = t + 1 (that is, YES), the process proceeds to step S110, and if it is determined that t ′ ≠ t + 1 (that is, NO), the process proceeds to step S113.

In steps S110 to S112, the first write buffer 5x is selected as a record write destination.
First, it is determined whether or not the first write buffer 5x is full (step S110). If it is determined that the first write buffer 5x is not full (that is, NO), the process immediately proceeds to step S112 and is full ( In other words, if it is determined as YES), the records in the first write buffer 5x are collectively added to the end of the bucket t + 1 to empty the first write buffer 5x (step S111).

  Subsequently, in step S112, the record updated in step S106 is added to the end of the first write buffer 5x, and the next update time 92 of the information source registration table 9b corresponding to the information source identifier of the record is set to “t + 1”. (= T ′) ”and the process returns to step S104.

On the other hand, in steps S113 to S115, the second write buffer 5y is selected as a record write destination.
First, it is determined whether the second write buffer 5y is full (full) (step S113). If it is determined that the second write buffer 5y is not full (that is, NO), the process immediately proceeds to step S115 and is full ( That is, if it is determined as YES), the records in the second write buffer 5y are sorted in the order of the next update time, and added to the end of the corresponding bucket at the same time for the next next update time. When the record has been written, the second write buffer is emptied (step S114).

  Subsequently, in step S115, the record updated in step S106 is added to the end of the second write buffer 5y, and the next update time 92 of the information source registration table corresponding to the information source identifier of the record is set to “t ′”. And return to step S104.

  On the other hand, if it is determined in step S104 that all record processing of the read buffer 4 has been completed (that is, YES), the first write buffer 5x is set as the “new read buffer”, and the original read buffer 4 is changed. It is emptied to be a “new first write buffer”, and the roles of the read buffer 4 and the first write buffer 5x are exchanged (step S116).

  Finally, as in step S114, the second write buffer 5y is flushed and emptied (step S117), the processed bucket t is deleted (step S118), and the process returns to step S101.

Next, the operation of the request reception unit 10A when receiving a registration request will be described with reference to FIG.
In FIG. 10, steps S201, S202 and S204 are the same processes as steps S21, S22 and S24 described above (see FIG. 4), respectively.

  First, when receiving the registration request in step S201, the request receiving unit 10A searches the information source registration table 9b in step S202. If the information source identifier R of the registration request is found, the request receiving unit 10A proceeds to step S205. If R is not found, the process proceeds to step S203.

In step S203, the record corresponding to the information source identifier R of the registration request is initialized, and the next (first time) update time t ′ at that time is determined. As described above, the next update time t ′ can be determined by adding a predetermined constant (for example, “1”) or a predetermined range of random numbers to the current time t.
Subsequently, in step S204, the information source identifier R is additionally registered in the information source registration table 9b, and the process proceeds to step S208.

  In step S208, as in steps S109 to S115 in FIG. 9, the information source identifier R corresponds to the end of the first or second write buffer 5x, 5y according to the value of the next update time t ′. Write the record and return to step S201.

  On the other hand, in step S205, it is determined whether or not the next update time 92 corresponding to the information source identifier R in the information source registration table 9b indicates a rejection bucket, and is determined to indicate the rejection bucket (that is, YES). If it is determined that the rejection bucket is not indicated (ie, NO), the process returns to step S201.

In step S206, the rejection bucket corresponding to the information source identifier R is scanned from the end, and the value at the time when the record corresponding to the information source identifier R is finally rejected is acquired.
Subsequently, as in step S203, the value of the next update time t ′ is determined (step S207), and the process proceeds to step S208.

Next, the operation of the request receiving unit 10A when a search request (including the information source identifier Q of the search target information source) is received will be described with reference to FIG.
In FIG. 11, step S211 is the same processing as step S201 described above (see FIG. 10).

  First, when receiving a search request in step S211, the request receiving unit 10A searches the information source registration table 9b for the information source identifier Q of the search request (step S212). If the information source identifier Q is found, step S213 is performed. If the information source identifier Q is not found, the process proceeds to step S216.

In step S213, the next update time 92 corresponding to the information source identifier Q is acquired from the information source registration table 9b and is set to “s”.
Subsequently, scanning is performed from the top in the order of the read buffer 4, the first write buffer 5x, the second write buffer 5y, and the bucket s, and a record corresponding to the information source identifier Q is acquired (step S214). .

However, if it is not the next update time s = current time t, the scanning process of the reading buffer 4 can be skipped in step S214.
Thereafter, the acquired record is transmitted to the request source (step S215), and the process returns to step S211.
On the other hand, in step S216, a search failure notification is transmitted to the request source, and the process returns to step S211.

Next, the operation of the request receiving unit 10A when receiving an update request (including the information source identifier U of the search target information source and the update content) will be described with reference to FIG.
In FIG. 12, step S221 is the same process as step S211 described above (see FIG. 11).

  First, when receiving the search request in step S221, the request receiving unit 10A searches the information source registration table 9b for the information source identifier U of the update request (step S222). If the information source identifier U is found, the request receiving unit 10A performs step S223. If the information source identifier Q is not found, the process returns to step S221.

In step S223, the next update time 92 corresponding to the information source identifier U is acquired from the information source registration table 9b and is set to “r”.
Subsequently, the update record 30 is configured with the information source identifier 31 as “U”, the next update time 32 as “r”, and the update content 37 as the update content included in the update request, in accordance with the value r of the next update time 32. Then, the data is written into an appropriate buffer (step S224).

That is, the update record 30 is added to the read buffer 4 if the next update time r = current time t, added to the first write buffer 5x if r = t + 1, and second if r> t + 1. To the write buffer 5y.
If the first write buffer 5x and the second write buffer 5y are full, flush processing is performed in the same manner as in steps S111 and S114 in FIG. Thereafter, the process returns to step S221.

  As described above, the periodic update data management system according to Embodiment 3 (FIG. 6) of the present invention includes the single read buffer 4 connected to the storage unit 1A, the first and the second connected to the storage unit 1A, and The update processing unit 6A includes all the second write buffers 5x and 5y, and reads all target records stored in the processing target bucket into the read buffer 4 at once, and for each target record in the read buffer 4, The updated information and the next update time are determined by referring to the target information source corresponding to the target record from the plurality of information sources.

  In addition, the update processing unit 6A determines that the next update time determined is a value larger by one unit than the current time t and a predetermined number of records are already stored in the first write buffer 5x. Writes all records in the first write buffer 5x to the end of the area corresponding to the next update time, emptyes the first write buffer 5x, and the determined next update time is greater than the current time. When the value is larger by one unit, the updated value of each record is stored at the end of the first write buffer 5x.

  In addition, the update processing unit 6A determines that the next update time determined is a value that is two units or more larger than the current time t, and a predetermined number of records are already stored in the second write buffer 5y. Writes all the records in the second write buffer 5y in the order of the next update time to the end of the area corresponding to each next update time of all the records to empty the second write buffer 5y. If the determined next update time is 2 units or more larger than the current time, the updated values of all the records are stored at the end of the second write buffer 5y and read. After all the records in the buffer 4 have been processed, the roles of the read buffer 4 and the first write buffer 5x are exchanged, and all the records in the second write buffer 5y are changed in the order of the next update time. all Writing at the end of the area corresponding to the next update time of each record, after the second write buffer 5y empty, delete the processed bucket.

  Thus, by providing the second write buffer 5y that temporarily stores records to be written in all buckets to be processed after the next time, the write buffer for each bucket as described above (FIG. 1). Compared to the case where 5a, 5b,... Are prepared, it is possible to prevent a decrease in storage efficiency due to a difference in usage rate for each write buffer, and to suppress the frequency of writing to the storage unit 1A (magnetic disk device). can do.

  Further, when the update processing unit 6A distributes and writes the records stored in the second write buffer 5y to the plurality of buckets 2, the records are rearranged and processed for each next update time, that is, for each bucket (step). S114, S117), performance close to sequential access can be obtained.

  In addition, a first write buffer 5x that temporarily stores records to be written in the bucket to be processed next time is provided, and the first write buffer 5x is set as a new read buffer when the processing target bucket is switched ( Since step S116), it is possible to reduce overhead due to writing to the next processing target bucket and rereading.

  According to Embodiment 3 of the present invention, the information source registration table 9b in the second storage unit 8A is an area for identifying a bucket (area) in which a record corresponding to each information source is stored. The update processing unit 6A and the request receiving unit 10A include identification information (next update time), and write a record corresponding to one of the plurality of information sources to one of the plurality of buckets (areas) 2 in the storage unit 1A. At this time, the area identification information corresponding to the record in the information source registration table 9b is updated.

  Further, the request accepting unit 10A accepts a search request including an identifier. When the identifier is stored in the information source registration table 9b, the request accepting unit 10A scans a bucket (area) corresponding to the identifier and corresponds to the identifier. The record to be read is read, and the contents of the record are returned to the requester of the search request.

  In this manner, by storing the next update time together with the information source identifier in the information source registration table 9b, the contents of the record corresponding to the information source can be searched without scanning all buckets.

  Further, according to the third embodiment of the present invention, the request receiving unit 10A receives an update request including an identifier corresponding to one of the plurality of information sources and an update content for the record corresponding to the information source, If the identifier is stored in the information source registration table 9b, the update request is written at the end of the bucket (area) corresponding to the identifier.

Further, the update processing unit 6A processes the records stored in the processing target bucket and the update request in the order of identifiers, and determines an updated value of the record based on the result of referring to the corresponding information source and the update request. To do.
In this way, in addition to the normal record 3, the update records 30 are stored in the buckets 2a to 2c, and the update processing unit 6A performs the processing (step S105), whereby the contents of the records corresponding to the information sources are stored. Can be updated externally.

  Further, according to Embodiment 3 of the present invention, the second storage unit 8A has the rejection buckets 12a and 12b for storing a plurality of records, and the update processing unit 6A has a predetermined value after updating the records. If the condition is satisfied, the record is written at the end of the rejection bucket, and the area identification information corresponding to the record in the information source registration table 9b is set in the rejection bucket.

  Further, the request receiving unit 10A scans the rejection bucket when the information source identifier of the received registration request is stored in the information source registration table 9b and the corresponding region identification information indicates the rejection bucket. The value of the record corresponding to the identifier is read, the record is reset based on the value of the record, and written to one of the plurality of buckets 2 or one end of the write buffer. The area identification information corresponding to the record of the source registration table 9b is updated.

  In this way, it has rejection buckets 12a, 12b,... And stores the value of the rejected record in the rejection bucket and records the storage location of the rejection record in the information source registration table 9b (step S108). Thus, when the request receiving unit 10A receives the registration request, the rejected record can be re-registered with the value at the time of rejection.

In the third embodiment, the first and second write buffers 5x and 5y are used. However, the first write buffer 5x is not used, and the second update time t + 1 is added to the second update buffer t. You may write in the write buffer (single write buffer) 5y.
In this case, the periodic update data management system includes a single read buffer 4 connected to the storage unit 1A and a single write buffer 5y connected to the storage unit 1A. All the target records stored in the target bucket are collectively read into the read buffer 4, and for each target record in the read buffer 4, refer to the target information source corresponding to the target record from the plurality of information sources, The updated value and the next update time are determined.

  Further, when a predetermined number of records are already stored in the write buffer 5y, the update processing unit 6A changes all the records in the write buffer 5y in the order of the next update time to the next of all the records. After writing at the end of the area corresponding to the update time and emptying the write buffer 5y, the updated value of the target record is stored at the end of the write buffer 5y.

  If the write buffer 5y is not empty after processing all the records in the read buffer 4, the update processing unit 6A deletes all the records in the write buffer 5y in the order of the next update time. Is written at the end of the area corresponding to the next update time of each record, the write buffer 5y is emptied, and then the processing target buffer is deleted.

  As a result, the effect of avoiding writing and reading to a part of the record of the bucket to be processed next time cannot be obtained, but write buffers 5a, 5b,... Are prepared for each bucket 2 as described above (FIG. 1). In this case, it is possible to prevent the storage efficiency from being lowered, and the effect of suppressing the frequency of writing to the storage unit 1A (magnetic disk device) is maintained.

  In the first to third embodiments, the unit of the next update time is fixed (for example, set to one day). For example, for updates within one day, for updates after one hour, one day or more. The unit may be variably set such as one day unit.

  In this way, by setting the unit corresponding to the time resolution of the next update time to a different value depending on the difference from the current time, many very small buckets are required while reducing the shortest update cycle. Therefore, the access efficiency to the storage unit 1A (magnetic disk) and the update speed can be improved at the same time.

Further, in the first to third embodiments, a magnetic disk device is used as the storage unit 1A and a Web page or file is used as the information sources 7a to 7c. However, a flash memory is used as the storage unit 1A and the information sources 7a to 7c are used. A sensor element having a communication function may be used.
For example, when sensor elements having a communication function are used as the information sources 7a to 7c, the update processing units 6 and 6A acquire data from the sensor elements corresponding to the target records in the processing target buffer, and update the target records. The subsequent values are determined, and the request receiving units 10 and 10A receive a registration request from the sensor element.

  In general, the flash memory is not suitable for pure random access, but access in units of blocks is required from the viewpoint of function and performance. Therefore, by using the blocks of the flash memory as the buckets 2a to 2c, these restrictions are imposed. Performance degradation due to can be avoided.

In this case, the final value 33 (see FIGS. 2 and 8) of the record 3 stores the data value itself acquired from the sensor element, and the update processing means 6 and 6A perform steps 5, S106 (FIG. 3, FIG. 3). 9), the next update time can be determined in proportion to the rate of change (per unit time) of the value of the final state 33.
Furthermore, the request receiving units 10 and 10A can cope with an environment in which the sensor element is additionally installed in an ad hoc by receiving a registration request from the sensor element itself.

It is a block block diagram which shows the periodic update data management system which concerns on Embodiment 1 of this invention. It is explanatory drawing which shows in detail the information contained in the record in the bucket by Embodiment 1 of this invention. It is a flowchart which shows operation | movement of the update process part by Embodiment 1 of this invention. It is a flowchart which shows operation | movement of the request | requirement reception part by Embodiment 1 of this invention. It is a flowchart which shows the principal part operation | movement of the update process part by Embodiment 2 of this invention. It is a block block diagram which shows the periodic update data management system which concerns on Embodiment 3 of this invention. It is explanatory drawing which shows the information source registration table by Embodiment 3 of this invention in detail. It is explanatory drawing which shows in detail the information contained in the record in the bucket by Embodiment 3 of this invention. It is a flowchart which shows operation | movement of the update process part by Embodiment 3 of this invention. It is a flowchart which shows operation | movement of the request | requirement reception part at the time of the registration request reception by Embodiment 3 of this invention. It is a flowchart which shows operation | movement of the request | requirement reception part at the time of the search request reception by Embodiment 3 of this invention. It is a flowchart which shows operation | movement of the request | requirement reception part at the time of the update request reception by Embodiment 3 of this invention.

Explanation of symbols

  1, 1A storage unit, 2, 2a to 2c bucket (area), 3 records, 4 read buffer, 5a, 5b write buffer, 5x first write buffer, 5y second write buffer, 6, 6A update Processing unit, 7a to 7c Information source, 8, 8A Second storage unit, 9a, 9b Information source registration table, 10, 10A Request receiving unit, 11 Control terminal, 12a, 12b Rejection bucket.

Claims (19)

A periodic update data management system comprising a storage unit and an update processing unit,
The storage unit stores a plurality of records corresponding one-to-one with a plurality of external information sources in a plurality of different areas based on respective next update times,
The update processing unit
Among the plurality of areas, the one corresponding to the current time is selected as a processing target area,
For the target record in the processing target area, refer to the target information source corresponding to the target record from the plurality of information sources, and after updating the target record based on the acquired information acquired from the target information source And determine the value of
Based on the presence or absence of the change from the previous reference time of the acquired information or the size, determine the next update time of the target record,
Store the updated value of the target record at the end of the area corresponding to the determined next update time,
The periodic update data management system, wherein after all the target records in the processing target area have been processed, the processing target area is deleted. A single read buffer connected to the storage unit;
A plurality of areas in the storage unit and a plurality of write buffers having a predetermined capacity corresponding to one-to-one;
The update processing unit
Read all target records stored in the processing target area into the reading buffer in a batch,
For each target record in the read buffer, referring to the target information source corresponding to the target record from among the plurality of information sources, determining the updated value and the next update time,
A first write buffer corresponding to the determined next update time is selected from the plurality of write buffers;
If a record of a predetermined capacity is already stored in the first write buffer,
Writing all records in the first write buffer to the end of the area corresponding to the first write buffer, emptying the first write buffer,
Storing the updated value of the target record at the end of the first write buffer;
After processing all records in the read buffer,
Next, all the next target records in the second write buffer corresponding to the next processing target area to be processed are written at the end of the next processing target area, and the second write buffer is emptied. From
The periodic update data management system according to claim 1, wherein the second write buffer and the processing target area are deleted. The update processing unit
After processing all records in the read buffer,
The second write buffer corresponding to the next processing target area as a new read buffer,
3. The periodic update data management system according to claim 2, wherein the original read buffer and the processing target area are deleted. A single read buffer connected to the storage unit;
A single write buffer connected to the storage unit,
The update processing unit
Read all target records stored in the processing target area into the reading buffer in a batch,
For each target record in the read buffer, referring to the target information source corresponding to the target record from among the plurality of information sources, determining the updated value and the next update time,
When a predetermined number of records are already stored in the write buffer,
Write all the records in the write buffer in the order of the next update time to the end of the area corresponding to the next update time of each of all the records, empty the write buffer,
Store the updated value of the target record at the end of the write buffer;
After processing all records in the read buffer,
If the write buffer is not empty,
Write all the records in the write buffer in the order of the next update time to the end of the area corresponding to the next update time of each of all the records, empty the write buffer,
The periodic update data management system according to claim 1, wherein the processing target area is deleted. A single read buffer connected to the storage unit;
First and second write buffers connected to the storage unit,
The update processing unit reads all target records stored in the processing target area into the reading buffer in a batch,
For each target record in the read buffer, referring to the target information source corresponding to the target record from among the plurality of information sources, determining the updated value and the next update time,
When the determined next update time is a value larger by one unit than the current time and a predetermined number of records are already stored in the first write buffer,
All records in the first write buffer are written at the end of the area corresponding to the next update time, the first write buffer is emptied, and the determined next update time is greater than the current time Is also a value larger by one unit,
Storing updated values of each of all the records at the end of the first write buffer;
When the determined next update time is a value larger by 2 units or more than the current time and a predetermined number of records are already stored in the second write buffer,
All the records in the second write buffer are written in the order of the next update time to the end of the area corresponding to the next update time of each of all the records, and the second write buffer is emptied. When the determined next update time is 2 units or more larger than the current time,
Storing updated values of each of all the records at the end of the second write buffer;
After processing all records in the read buffer,
Exchanging roles of the read buffer and the first write buffer;
All the records in the second write buffer are written in the order of the next update time to the end of the area corresponding to the next update time of each of all the records, and the second write buffer is emptied. After
The periodic update data management system according to claim 1, wherein the processing target area is deleted.   The unit corresponding to the time resolution of the next update time is set to a different value in accordance with the magnitude of the difference from the current time. The periodic update data management system described. The update processing unit
When the updated value of the record satisfies a predetermined condition,
The periodic update data management system according to any one of claims 1 to 6, wherein the record is not written to any of the plurality of areas or the write buffer. A second storage unit different from the storage unit;
A request receiving unit connected to the second storage unit,
The second storage unit has an information source registration table that stores identifiers of information sources corresponding to all records stored in the storage unit,
The request receiving unit
When a registration request including an identifier corresponding to any one of the plurality of information sources is received and the identifier is not stored in the information source registration table, the identifier is added to the information source registration table. ,
It initializes the record corresponding to the identifier to a predetermined value, claims 1 to 7 in which one of a plurality of regions, or to write to one end of the write buffer, wherein in said storage unit The periodic update data management system according to any one of the preceding items. The information source registration table includes area identification information for identifying an area in which a record corresponding to each information source is stored,
The update processing unit and the request receiving unit are:
When writing a record corresponding to any of the plurality of information sources to any of the plurality of areas in the storage unit, update the area identification information corresponding to the record of the information source registration table,
The request receiving unit
When a search request including the identifier is received and the identifier is stored in the information source registration table, the area corresponding to the identifier is scanned to read the record corresponding to the identifier, and the content of the record is 9. The periodic update data management system according to claim 8 , wherein a reply is made to the request source of the search request. The request receiving unit
Receiving an update request including an identifier corresponding to one of the plurality of information sources and an update content for a record corresponding to the information source;
When the identifier is stored in the information source registration table,
Write an update request at the end of the area corresponding to the identifier,
The update processing unit processes records and update requests stored in the processing target area in the order of the identifiers,
10. The periodic update data management system according to claim 9 , wherein the updated value of the record is determined based on the result of referring to the corresponding information source and the update request. The update processing unit
When the updated value of the record satisfies a predetermined condition,
Not writing the record to any of the plurality of regions or the write buffer;
The periodic update data management system according to any one of claims 8 to 10 , wherein an identifier corresponding to the record is deleted from the information source registration table. The second storage unit has a rejection area for storing a plurality of records,
The update processing unit
When the updated value of the record satisfies a predetermined condition,
Write the record at the end of the rejection area,
Set the area identification information corresponding to the record of the information source registration table in the rejection area,
The request receiving unit
When the information source identifier of the accepted registration request is stored in the information source registration table and the corresponding area identification information indicates the rejection area,
The rejection area is scanned to read the value of the record corresponding to the identifier, the record is reset based on the value of the record, and one of the plurality of areas or one of the write buffers Write at the end of
The period update data management system according to claim 9 or 10 , wherein region identification information corresponding to a record of the information source registration table is updated. The update processing unit
When a predetermined number of records in the processing target area have been processed, or when a predetermined time has passed since the processing start of the processing target area,
Without referring to an information source corresponding to an unprocessed record, the next update time is updated to a predetermined value and written to one end of the plurality of areas or one end of the write buffer. The periodic update data management system according to any one of claims 1 to 12, characterized in that:   The periodic update data management system according to claim 13, wherein the update processing unit processes each record of the processing target area in order of priority.   The periodic update data management system according to any one of claims 1 to 14, wherein the storage unit is a magnetic disk device.   The periodic update data management system according to any one of claims 1 to 14, wherein the storage unit is a flash memory. The information source is a web page on a website,
The identifier of the information source is a URL of a web page,
The update processing unit
Download the contents of the web page corresponding to the target record in the processing target area, determine the updated value of the target record,
Extracting a link from a Web page corresponding to the target record any one of claims 8, characterized in that sending said request receiving unit each of the resulting URL as registration request to claim 12 The periodic update data management system described in 1. The information source is a file on a file server;
The identifier of the information source is a path name of the file,
The update processing unit
Download the contents of the file corresponding to the target record in the processing target area, determine the updated value of the target record,
Wherein when a file that corresponds to the target record is a directory, claim from claim 8, characterized in that sending said request receiving unit each path name of the file and subdirectories in the directory as a registration request 12 The periodic update data management system according to any one of the preceding items. The information source is a sensor element having a communication function,
The update processing unit obtains data from a sensor element corresponding to a target record in the processing target region, determines an updated value of the target record,
The periodic update data management system according to any one of claims 8 to 12, wherein the request reception unit receives a registration request from the sensor element.

Images