JP7634259B2 - Network scanning device, a program for causing a computer to execute the program, and a computer-readable recording medium having the program recorded thereon - Google Patents

Description

Article 30, paragraph 2 of the Patent Act applies. Published in Proceedings of the 2019 Institute of Electronics, Information and Communication Engineers Society Conference, Communications Lecture Papers 2, BS-4-2, pages S-14 to S-15 (published August 27, 2019)

Application of Article 30, Paragraph 2 of the Patent Act Published at the 2020 IEEE 17th Annual Consumer Communications & Networking Conference (CCNC) (published on January 10, 2020)

This invention relates to a network scanning device, a program for causing a computer to execute the program, and a computer-readable recording medium on which the program is recorded.

As the fifth generation mobile communication (5G) system comes into practical use, many devices, such as mobile terminals, computers, sensors, and webcams, will be connected to the Internet. Such mobile terminals and computers connected to the Internet are called IoT (Internet of Things) devices.

In the IoT era, the number of IoT devices connected to communication networks is expected to increase rapidly. Among these, IoT devices that perform wireless communication are expected to be used illegally without sufficient security measures in order to increase the number of cases where they are exploited in cyber attacks such as distributed denial of service (DDoS) attacks.

To avoid such a situation, it is necessary to conduct network scans of IoT devices connected to communication networks and investigate whether security measures are in place for each IoT device.

The network scan described in Non-Patent Document 1 is known. The network scan described in Non-Patent Document 1 scans applications in the IPv4 space, and is performed by the following method.

First, a port scan is performed in the IPv4 space to check for responses. Then, for hosts that respond, a handshake is performed using various protocols (HTTP (Hypertext Transfer Protocol), HTTPS (Hypertext Transfer Protocol Secure), POP3 (Post Office Protocol version 3), etc.) to obtain messages. Keywords that are useful for vulnerability assessment are then extracted from the obtained messages. Metadata and tag information are then assigned to the keywords. The keywords with the assigned metadata and tag information are then registered in the database as structured data.

Z. Durumeric, et al., “A Search Engine Backed by Internet-Wide Scanning”, CCS15, October 12-16, 2015. Katsunari Yoshioka, Kosuke Murakami, Tsutomu Matsumoto, "Network Scanning Method for Detecting Malware-Infected Hosts and Automatic Generation of Detection Signatures," Transactions of the Information Processing Society of Japan, Vol. 51, No. 9, pp. 1633-1644 (Sep. 2010).

In a network scan, the round-trip time (RTT), which is the time between sending a scan packet and receiving a scan response, has very similar characteristics for terminal devices when the network is congested or when communication quality is insufficient.

As a result, when a network scan is performed in an actual communication environment, there is an increased rate at which the cause of the network scan failure is mistakenly assumed to be insufficient communication quality, even though the cause is in fact network congestion.

Therefore, according to an embodiment of the present invention, a network scanning device is provided that can accurately estimate the results of a network scan performed in an actual communication environment.

In addition, according to an embodiment of the present invention, a program is provided that causes a computer to perform an accurate estimation of the results of a network scan performed in an actual communication environment.

Furthermore, according to an embodiment of the present invention, a computer-readable recording medium is provided that stores a program for causing a computer to perform an accurate estimation of the results of a network scan performed in an actual communication environment.

(Configuration 1)
According to an embodiment of the present invention, a network scanning device includes a scanning means, a detecting means, a generating means, an updating means, and an estimating means. The scanning means executes a network scan in a first communication environment where the results of a network scan for M (M is an integer equal to or greater than 2) terminal devices are unknown, and acquires first scan response delays of the network scan for all of the M terminal devices. The detecting means executes a detection process to detect, as similar scan response delays, M' second scan response delays that are most similar to the M first scan response delays among a plurality of sets of M'(M' is an integer equal to or greater than 2) second scan response delays obtained when the results of a network scan executed in a second communication environment where the results of the network scan are known are a plurality of known estimated results. The generating means acquires, for all of the M' second scan response delays, an optimal communication characteristic that is the communication characteristic most suitable for estimating a known estimation result among a plurality of communication characteristics acquired together with one of the M' second scan response delays constituting the similar scan response delay, and generates similar standard sample data consisting of the acquired M' optimal communication characteristics and the M' second scan response delays. The updating means trains an estimation algorithm that estimates a result of a network scan performed in the first communication environment based on the similar standard sample data, and updates the estimation algorithm using an evaluation index so as to approach the optimal algorithm. The estimation means estimates a result of a network scan performed in the first communication environment using the estimation algorithm updated by the updating means. The optimal algorithm is an algorithm that estimates a result of a network scan such that the estimated result when the result of the network scan is estimated matches a known estimation result. The evaluation index is an index that evaluates the accuracy of an estimation result when the result of a network scan is estimated by the trained algorithm.

(Configuration 2)
In configuration 1, the generating means further generates training data and test data from the similar standard sample data and generates a training set and a validation set from the training data, generates initial sample data by combining a portion of each of a plurality of standard sample data corresponding to a plurality of known estimation results and each of which consists of M′ second scan response delays and M′ optimal communication characteristics, and generates provisional sample data by replacing a portion of the initial sample data consisting of the same type of standard sample data as the training set with a portion of the training set. The updating means updates the estimation algorithm with the provisional sample data when a first evaluation index for evaluating an estimation result obtained by training the estimation algorithm using the initial sample data and estimating the result of the network scan using the trained estimation algorithm using the validation set is equal to or smaller than a second evaluation index for evaluating an estimation result obtained by training the estimation algorithm using the provisional sample data and estimating the result of the network scan using the trained estimation algorithm using the validation set.

(Configuration 3)
In configuration 2, the multiple known estimation results include a first estimation result indicating that the result of the network scan is normal, a second estimation result indicating that the network scan has failed due to insufficient communication quality, and a third estimation result indicating that the network scan has failed due to network congestion. The multiple standard sample data include first standard sample data, second standard sample data, and third standard sample data. The first standard sample data is standard sample data consisting of M' second scan response delays and M' optimal communication characteristics acquired in a communication environment in which the network is normal, and is used to estimate that the result of the network scan is the first estimation result. The second standard sample data is standard sample data consisting of M' second scan response delays and M' optimal communication characteristics acquired in a communication environment in which the network communication quality is insufficient, and is used to estimate that the result of the network scan is the second estimation result. The third standard sample data is standard sample data consisting of M′ second scan response delays and M′ optimal communication characteristics acquired in a communication environment with a congested network, and is used to estimate that the result of the network scan is the third estimated result. The first estimated result is a result estimated by an estimation algorithm using the first standard sample data, the second estimated result is a result estimated by an estimation algorithm using the second standard sample data, and the third estimated result is a result estimated by an estimation algorithm using the third standard sample data.

(Configuration 4)
In the configuration 2 or 3, the generating means generates a plurality of provisional sample data constituting the provisional sample data by executing, for all of the plurality of subset data, replacing a part of sample data of the initial sample data consisting of standard sample data of the same type as that of the training set with one of the plurality of subset data of the training set. The updating means executes, for all of the plurality of provisional sample data, a first process of training an estimation algorithm using the initial sample data and generating a first confusion matrix based on an estimation result obtained when a result of a network scan is estimated by the estimation algorithm after training using the validation set, a second process of training an estimation algorithm using one of the plurality of provisional sample data constituting the provisional sample data and generating a second confusion matrix based on an estimation result obtained when a result of a network scan is estimated by the estimation algorithm after training using the validation set, and a third process of updating the estimation algorithm with optimal sample data consisting of one provisional sample data among the provisional sample data when the maximum evaluation index of the second evaluation index is obtained, which is greater than the first evaluation index, based on a plurality of second evaluation indexes calculated based on the second confusion matrix; a fourth process of generating a third confusion matrix based on an estimation result obtained when the result of the network scan is estimated by the updated estimation algorithm using test data after the third process; and a fifth process of approving the updated estimation algorithm and the optimal sample data when the third evaluation index calculated based on the third confusion matrix is equal to or greater than the maximum evaluation index.

(Configuration 5)
In configuration 4, when the third evaluation index is smaller than the maximum evaluation index, the update means repeatedly executes the first process to the fifth process using the estimation algorithm updated with the optimal sample data until the third evaluation index becomes equal to or larger than the maximum evaluation index.

(Configuration 6)
In the configuration 4 or 5, the network scanning device further includes a calculation means for calculating an evaluation index. Each of the first to third confusion matrices includes a first element indicating a rate of correctly estimating the occurrence of an assumed event when the assumed event occurs, a second element indicating a rate of correctly estimating that the assumed event did not occur when the assumed event did not actually occur, a third element indicating a rate of erroneously estimating the occurrence of the assumed event as another event when the assumed event occurs, and a fourth element indicating a rate of erroneously estimating that the assumed event occurred when the assumed event did not actually occur. Then, the calculation means calculates Precision, which is the ratio of the proportion actually classified into the first element among the proportion classified into the first element or the fourth element, and Recall, which is the ratio of the proportion actually classified into the first element among the proportion classified into the first element or the third element, and executes a third calculation process of calculating 2×Precision/(Precision+Recall) based on Precision and Recall based on the first confusion matrix to calculate a first evaluation index, executes the third calculation process based on a plurality of second confusion matrices to calculate a plurality of second evaluation indexes, and executes the third calculation process based on the third confusion matrix to calculate a third evaluation index.

(Configuration 7)
In the configuration 4 or 5, the network scanning device further includes a calculation means for calculating an evaluation index. Each of the first to third confusion matrices includes a first element indicating a rate of correctly estimating the occurrence of an assumed event when the assumed event occurs, a second element indicating a rate of correctly estimating that the assumed event did not occur when the assumed event did not actually occur, a third element indicating a rate of erroneously estimating the occurrence of the assumed event as another event when the assumed event occurs, and a fourth element indicating a rate of erroneously estimating that the assumed event occurred when the assumed event did not actually occur. Then, the calculation means calculates a first evaluation index consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the first confusion matrix, calculates a plurality of second evaluation indexes consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the plurality of second confusion matrices, and calculates a third evaluation index consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the third confusion matrix.

(Configuration 8)
In any of configurations 1 to 7, the detection means, in a detection process, detects M' second scan response delays having a distribution most similar to the distribution of the M first scan response delays from the multiple sets of M' second scan response delays as similar scan response delays based on multiple distributions of the multiple sets of M' second scan response delays and the distribution of the M first scan response delays.

(Configuration 9)
In configuration 8, the detection means performs a first calculation process in the detection process to calculate a first mode indicating a value of the first scan response delay that is most likely to occur based on M first scan response delays, and calculates a first mean absolute deviation that is an average value of absolute values of differences between the M first scan response delays and the first mode, and after the first calculation process, calculates a second mode indicating a value of the second scan response delay that is most likely to occur based on M' second scan response delays, calculates a second mean absolute deviation that is an average value of absolute values of differences between the M' second scan response delays and the second mode, calculates a difference in mean absolute deviation consisting of the absolute value of the difference between the first mean absolute deviation and the second mean absolute deviation, and subtracts the difference in mean absolute deviation from 1 to calculate a similarity, for all of the multiple known estimation results to obtain multiple similarities, and detects the M' second scan response delays that were used to calculate the highest similarity among the multiple obtained similarities as the similar scan response delay.

(Configuration 10)
According to an embodiment of the present invention, the program further comprises:
A first step in which a scanning means executes a network scan in a first communication environment in which the results of a network scan are unknown for M (M is an integer equal to or greater than 2) terminal devices, and acquires a first scan response delay of the network scan for each of the M terminal devices;
a second step of executing a detection process in which the detection means detects, as similar scan response delays, M' second scan response delays most similar to the M first scan response delays among a plurality of sets of M'(M' is an integer equal to or greater than 2) second scan response delays obtained when the results of a network scan executed in a second communication environment in which the results of the network scan are known are a plurality of known estimated results;
a third step in which the generating means acquires, for each of the M' second scan response delays, an optimal communication characteristic that is the communication characteristic most suitable for estimating a known estimation result among a plurality of communication characteristics acquired together with one of the M' second scan response delays constituting the similar scan response delay, and generates similar standard sample data consisting of the acquired M' optimal communication characteristics and the M' second scan response delays;
a fourth step in which an updating means trains an estimation algorithm for estimating a result of a network scan performed in the first communication environment based on similar standard sample data, and updates the estimation algorithm using an evaluation index so as to approach an optimal algorithm;
a fifth step in which the estimation means estimates a result of the network scan performed in the first communication environment using the estimation algorithm updated in the fourth step;
The optimal algorithm is an algorithm that estimates the results of a network scan such that the estimated results match known estimated results, and the evaluation index is an index that evaluates the accuracy of the estimated results when the results of a network scan are estimated by the trained algorithm.

(Configuration 11)
In configuration 10, in the third step, the generating means further generates training data and test data from the similar standard sample data, and generates a training set and a validation set from the training data, generates initial sample data by combining a portion of each of a plurality of standard sample data corresponding to a plurality of known estimation results and each of which is composed of M' second scan response delays and M' optimal communication characteristics, and generates provisional sample data by replacing a portion of the initial sample data composed of standard sample data of the same type as the training set with a portion of the training set;
In the fourth step, the updating means updates the estimation algorithm with the hypothetical sample data when a first evaluation index, which evaluates an estimation result when the estimation algorithm is trained using initial sample data and the network scan result is estimated by the trained estimation algorithm using a validation set, is equal to or less than a second evaluation index, which evaluates an estimation result when the estimation algorithm is trained using hypothetical sample data and the network scan result is estimated by the trained estimation algorithm using a validation set.

(Configuration 12)
In configuration 11, the plurality of known estimated results include a first estimated result indicating that the result of the network scan is normal, a second estimated result indicating that the network scan has failed due to insufficient communication quality, and a third estimated result indicating that the network scan has failed due to network congestion. The plurality of standard sample data include a first standard sample data, a second standard sample data, and a third standard sample data.

The first standard sample data consists of M' second scan response delays and M' optimal communication characteristics acquired in a communication environment in which the network is normal, and is standard sample data for estimating that the result of the network scan is the first estimated result.

The second standard sample data consists of M' second scan response delays and M' optimal communication characteristics obtained in a communication environment where the network communication quality is insufficient, and is standard sample data for estimating that the result of the network scan is the second estimated result.

The third standard sample data is composed of M' second scan response delays and M' optimal communication characteristics obtained in a communication environment with a congested network, and is standard sample data for estimating that the result of the network scan is the third estimated result.

The first estimation result is a result estimated by an estimation algorithm using the first standard sample data, the second estimation result is a result estimated by an estimation algorithm using the second standard sample data, and the third estimation result is a result estimated by an estimation algorithm using the third standard sample data.

(Configuration 13)
In the configuration 11 or 12, in the third step, the generating means generates a plurality of provisional sample data constituting the provisional sample data by executing the replacement of a part of sample data of the initial sample data, which is made of standard sample data of the same type as the training set, with one of the plurality of subset data of the training set for all of the plurality of subset data;
The fourth step is:
A first sub-step in which an update means trains an estimation algorithm using initial sample data and generates a first confusion matrix based on an estimation result when the estimation algorithm after training uses a validation set to estimate a result of a network scan;
a second sub-step in which the update means trains the estimation algorithm using a first set of provisional sample data among a plurality of provisional sample data constituting the provisional sample data, and generates a second confusion matrix based on an estimation result obtained when the result of the network scan is estimated by the trained estimation algorithm using a validation set;
a third sub-step of an update means for setting the second evaluation index to the first evaluation index and updating the estimation algorithm with the first provisional sample data when the second evaluation index calculated based on the second confusion matrix is greater than the first evaluation index calculated based on the first confusion matrix;
a fourth sub-step in which the update means does not set the second evaluation index to the first evaluation index when the second evaluation index is not greater than the first evaluation index, and selects second provisional sample data different from the first provisional sample data from the plurality of provisional sample data;
a fifth substep in which an updating means repeatedly executes the second to fourth substeps for all of the plurality of virtual sample data;
a sixth sub-step of generating a third confusion matrix based on an estimation result when the update means estimates the result of the network scan by the estimation algorithm finally updated in the fifth sub-step using the test data;
The seventh sub-step includes an updating means for approving the finally updated estimation algorithm when the third evaluation index calculated based on the third confusion matrix is equal to or greater than the first evaluation index finally obtained in the fifth sub-step, and the first provisional sample data when the second evaluation index constituting the finally obtained first evaluation index is obtained.

(Configuration 14)
In configuration 13, the fourth step further includes an eighth substep in which, when the third evaluation index is smaller than the first evaluation index finally obtained in the fifth substep, the updating means repeatedly executes the second substep to the seventh substep using the finally updated estimation algorithm until the third evaluation index becomes equal to or larger than the first evaluation index finally obtained in the fifth substep.

(Configuration 15)
In the configuration 13 or 14, the calculation means further causes the computer to execute a sixth step of calculating an evaluation index;
Each of the first to third confusion matrices includes a first element indicating a rate at which the occurrence of an assumed event is correctly estimated when the assumed event occurs, a second element indicating a rate at which the occurrence of an assumed event is correctly estimated when the assumed event does not actually occur, a third element indicating a rate at which the occurrence of an assumed event is erroneously estimated as another event when the assumed event occurs, and a fourth element indicating a rate at which the occurrence of an assumed event is erroneously estimated when the assumed event does not actually occur.
In the sixth step, the calculation means calculates Precision, which is a ratio of the ratio actually classified into the first element among the ratios classified into the first element or the fourth element, and Recall, which is a ratio of the ratio actually classified into the first element among the ratios classified into the first element or the third element, and calculates 2×Precision/(Precision+Recall) based on Precision and Recall based on the first confusion matrix to calculate a first evaluation index, performs the calculation process based on a plurality of second confusion matrices to calculate a plurality of second evaluation indexes, and performs the calculation process based on a third confusion matrix to calculate a third evaluation index (Configuration 16).
In the configuration 13 or 14, the calculation means further causes the computer to execute a sixth step of calculating an evaluation index;
Each of the first to third confusion matrices includes a first element indicating a rate at which the occurrence of an assumed event is correctly estimated when the assumed event occurs, a second element indicating a rate at which the occurrence of an assumed event is correctly estimated when the assumed event does not actually occur, a third element indicating a rate at which the occurrence of an assumed event is erroneously estimated as another event when the assumed event occurs, and a fourth element indicating a rate at which the occurrence of an assumed event is erroneously estimated when the assumed event does not actually occur.
In the sixth step, the calculation means calculates a first evaluation index consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the first confusion matrix, calculates a plurality of second evaluation indexes consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the plurality of second confusion matrices, and calculates a third evaluation index consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the third confusion matrix.

(Configuration 17)
In any of configurations 10 to 16, the detection means, in the second step detection process, detects M' second scan response delays having a distribution most similar to the distribution of the M first scan response delays from the multiple sets of M' second scan response delays as similar scan response delays based on multiple distributions of the multiple sets of M' second scan response delays and the distribution of the M first scan response delays.

(Configuration 18)
In configuration 17, in the detection process of the second step, the detection means performs a first calculation process to calculate a first mode indicating a value of the first scan response delay that is most likely to occur based on M first scan response delays and calculate a first mean absolute deviation that is an average value of absolute values of differences between the M first scan response delays and the first mode, and after the first calculation process, calculates a second mode indicating a value of the second scan response delay that is most likely to occur based on M' second scan response delays, calculates a second mean absolute deviation that is an average value of absolute values of differences between the M' second scan response delays and the second mode, calculates a difference in mean absolute deviation consisting of the absolute value of the difference between the first mean absolute deviation and the second mean absolute deviation, and subtracts the difference in mean absolute deviation from 1 to calculate a similarity, for all of the multiple known estimation results to obtain multiple similarities, and detects the M' second scan response delays that were the basis for calculating the highest similarity among the multiple obtained similarities as similar scan response delays.

(Configuration 19)
Furthermore, according to an embodiment of the present invention, a recording medium is a computer-readable recording medium having the program according to any one of configurations 10 to 18 recorded thereon.

The results of network scans performed in a real communication environment can be accurately estimated.

1 is a schematic diagram of a communication system according to an embodiment of the present invention. FIG. 2 is a schematic diagram of the network scanning device shown in FIG. 1 . 3 is a conceptual diagram of a correspondence table TBL1 stored in the host information database shown in FIG. 2. 3 is a conceptual diagram of another correspondence table TBL2 stored in the host information database shown in FIG. 2. FIG. 4 is a conceptual diagram of yet another correspondence table TBL3 stored in the host information database shown in FIG. 3. FIG. 2 is a conceptual diagram of a scan schedule. FIG. 1 illustrates the time dependency of traffic. FIG. 13 is a diagram illustrating a correlation coefficient of communication characteristics. FIG. 13 is a diagram illustrating another correlation coefficient of communication characteristics. FIG. 13 is a diagram showing still another correlation coefficient of communication characteristics. FIG. 13 is a diagram showing still another correlation coefficient of communication characteristics. FIG. 13 is a diagram showing an example of original data when selecting a communication characteristic CMbest. FIG. 13 is a diagram for explaining a method for selecting a communication characteristic CMbest. FIG. 11 is a diagram for explaining a method for calculating a score of a communication characteristic. FIG. 13 is a diagram showing scores of communication characteristics. FIG. 13 is a diagram showing the calculation results of rank weight R _k . FIG. 13 is a diagram showing a specific distribution of scan response delays. FIG. 13 is a diagram showing a specific distribution of scan response delays. FIG. 13 is a diagram showing a specific distribution of scan response delays. FIG. 13 shows the correspondence between PDF regions and the number of samples generated. FIG. 13 is a diagram showing an example distribution of a probability density function of a scan response delay. FIG. 13 illustrates another exemplary distribution of scan response delays. FIG. 1 shows data when training the estimation algorithm ALG. FIG. 1 is a conceptual diagram of sample data when training the estimation algorithm ALG. FIG. 13 is a conceptual diagram showing updating of virtual sample data for modeling. FIG. 1 is a conceptual diagram of training and evaluation of the estimation algorithm ALG. FIG. 26 is a diagram showing a specific example of the initial sample data D_sample_initial for modeling shown in FIG. 25. FIG. 24 is a diagram showing a specific example of the training set shown in FIG. 23. FIG. 24 is a diagram showing a specific example of the validation set shown in FIG. 23. FIG. 24 is a diagram showing a specific example of the test data shown in FIG. 23. FIG. 1 is a conceptual diagram of a confusion matrix. FIG. 13 is a diagram for explaining a method for generating a confusion matrix. 10 is a flowchart illustrating a network scan operation. 34 is a first flowchart for explaining a detailed operation of step S10 in FIG. 33. 34 is a second flowchart for explaining the detailed operation of step S10 in FIG. 33. 34 is a first flowchart for explaining a detailed operation of step S11 in FIG. 33. 34 is a second flowchart for explaining the detailed operation of step S11 in FIG. 33. 34 is a third flowchart for explaining the detailed operation of step S11 in FIG. 33. 37 is a flowchart for explaining a detailed operation of step S130 in FIG. 36. 38 is a first flowchart for explaining a detailed operation of step S143 in FIG. 37. 38 is a second flowchart for explaining the detailed operation of step S143 in FIG. 37. 38 is a third flowchart for explaining the detailed operation of step S143 in FIG. 37. 41 is a flowchart for explaining detailed operations of step S1432 in FIG. 40. 41 is a flowchart for explaining detailed operations of step S1436 in FIG. 40. 38 is a flowchart for explaining detailed operations of step S145 in FIG. 37. 38 is a flowchart for explaining a detailed operation of step S145 in FIG. 37 when the Matthews correlation coefficient MCC is calculated in step S145. FIG. 13 is a diagram illustrating an example of a confusion matrix. FIG. 13 is a diagram showing simulation results of D-score. FIG. 13 illustrates the probability density function of the scan response delay RTT _m .

The embodiment of the present invention will be described in detail with reference to the drawings. Note that the same or corresponding parts in the drawings are given the same reference numerals and their description will not be repeated.

1 is a schematic diagram of a communication system according to an embodiment of the present invention. Referring to FIG. 1, a communication system 10 according to the embodiment of the present invention includes a network scanning device 1, base stations BS1 and BS2, and an Internet Service Provider (ISP).
) and terminal devices TM1 to TM18.

The network scanning device 1, base stations BS1 and BS2, the Internet provider ISP, and terminal devices TM1 to TM18 are placed in the communication space.

Base station BS1 is a base station of wireless communication network CNW1, base station BS2 is a base station of wireless communication network CNW2, and the Internet provider ISP is a communications carrier of wired communication network CNW3.

The wireless communication networks CNW1 and CNW2 are communication networks that perform wireless communication according to mutually different wireless communication standards.

In this embodiment of the invention, each of the terminal devices TM1 to TM13 is a wireless terminal, and each of the terminal devices TM14 to TM18 is a wired terminal.

The terminal devices TM1 to TM6 are located outside the wireless communication network CNW2 and inside the wireless communication network CNW1.

The terminal device TM7 is located within the overlapping area of the wireless communication networks CNW1 and CNW2.

Terminal devices TM8 to TM13 are located outside wireless communication network CNW1 and inside wireless communication network CNW2.

Terminal devices TM14 to TM18 are placed within the wired communications network CNW3 and are connected to the Internet provider ISP via wires.

Each of the terminal devices TM1 to TM13 is equipped with two wireless communication modules capable of wireless communication with base stations BS1 and BS2. Therefore, each of the terminal devices TM1 to TM13 is capable of wireless communication with base station BS1 or base station BS2.

Terminal devices TM1 to TM6 communicate wirelessly with base station BS1, terminal device TM7 communicates wirelessly with base station BS1 or base station BS2, and terminal devices TM8 to TM13 communicate wirelessly with base station BS2.

Terminal devices TM14 to TM18 communicate via wired connection via an Internet provider (ISP).

The network scanning device 1 performs network scanning on the terminal devices TM1 to TM18 using the method described below.

Figure 2 is a schematic diagram of the network scanning device 1 shown in Figure 1. Referring to Figure 2, the network scanning device 1 includes a network scanner 11, a scan analysis manager 12, a host information database 13, an extraction means 14, a calculation means 15, an estimation means 16, and a scan scheduler 17.

The network scanner 11 receives scan timing from the scan scheduler 17, and based on the received scan timing, performs a network scan on the ports of the terminal devices connected to the network 20 to be scanned, for example, using the well-known Nmap (see non-patent document 2).

Then, the network scanner 11 receives a scan response of the network scan from a terminal device connected to the network 20, and outputs the received scan response to the user (comprising network security, etc.) 30 and the scan analysis manager 12.

In addition, when a network scan is performed without monitoring background traffic, the network scanner 11 measures the scan response delay RTT _m (m is an integer satisfying 1≦m≦M, M is the total number of terminal devices TM1 to TM18 and is an integer of 2 or more), the packet loss rate PLR _m , the maximum scan delay DLmax _m and the minimum scan delay DLmin _m , and outputs the measured scan response delay RTT _m , the packet loss rate PLR _m , the maximum scan delay DLmax _m and the minimum scan delay DLmin _m to the scan analysis manager 12. That is, when a network scan is performed without monitoring background traffic, the network scanner 11 measures communication characteristics CM1 = [scan response delay RTT _m / packet loss rate PLR _m / maximum scan delay DLmax _m / minimum scan delay DLmin _m ] consisting of scan response delay RTT _m , packet loss rate PLR _m , maximum scan delay DLmax _m and minimum scan delay DLmin _m , and outputs the measured communication characteristics CM1 = [scan response delay RTT _m / packet loss rate PLR _m / maximum scan delay DLmax _m / minimum scan delay DLmin _m ] to the scan analysis manager 12.

Furthermore, when a network scan is performed by monitoring background traffic, the network scanner 11 measures the scan response delay RTT _m , packet loss rate PLR _m , delay DL _m , jitter JT _m , background maximum delay BGDLmax _m , and background minimum delay BGDLmin _m , and outputs the measured scan response delay RTT _m , packet loss rate PLR _m , delay DL _m , jitter JT _m , background maximum delay BGDLmax _m , and background minimum delay BGDLmin _m to the scan analysis manager 12. That is, when a network scan is performed by monitoring background traffic, the network scanner 11 measures communication characteristics CM2 = [scan response delay RTT _m / packet loss rate PLR _m / delay DL _m / jitter JT _m / background maximum delay BGDLmax _m / background minimum delay BGDLmin _m ] consisting of scan response delay RTT _m , packet loss rate PLR _m , delay DL _m , jitter JT _m , background maximum delay BGDLmax _m , and background minimum delay BGDLmin _m , and outputs the measured communication characteristics CM2 = [scan response delay RTT _m / packet loss rate PLR _m / delay DL _m / jitter JT _m / background maximum delay BGDLmax _m / background minimum delay BGDLmin _m ] to the scan analysis manager 12.

The scan analysis manager 12 receives a scan response from the network scanner 11 and receives a communication log during the network scan. The scan analysis manager 12 also receives communication characteristics CM1 or CM2 from the network scanner 11.

Then, when the scan analysis manager 12 receives the scan response and the communication characteristics CM1, it generates a scan result SC1 that associates the address of the terminal device to be scanned, the scan response, and the communication characteristics CM1 based on the scan response and the communication characteristics CM1, and creates or updates the host information database 13 by storing the generated scan result SC1 in the host information database 13.

In addition, when the scan analysis manager 12 receives the scan response and communication characteristics CM2, it generates a scan result SC2 that associates the address of the terminal device to be scanned, the scan response, and the communication characteristics CM2 based on the scan response and communication characteristics CM2, and creates or updates the host information database 13 by storing the generated scan result SC2 in the host information database 13.

The host information database 13 prestores in association with a scan response delay {RTT _{m' _known}_1 (m' = 1' to M') acquired when a network scan is performed in a communication environment where the result of the network scan is "normal" and a communication characteristic {CMbest m '} _known}_1 (m' = 1' to M') that is most suitable for estimating that the result of the network scan is "normal". Here, m' is an integer that satisfies 1'≦m'≦M', and M' is the total number of terminal devices that are targets of a network scan performed in a communication environment where the result of the network scan is known. Furthermore, m' may be the same as m or may be different from m.

In addition, the host information database 13 pre-stores in association with the scan response delay {RTT _{m' _known}_2 (m' = 1' to M') acquired when a network scan is performed in a communication environment where the network scan fails due to "insufficient communication quality", and the communication characteristic {CMbest m '} _known}_2 (m' = 1' to M') that is most suitable for inferring that the network scan has failed due to "insufficient communication quality".

Furthermore, the host information database 13 pre-stores in association with the scan response delay {RTT _{m' _known}_3 (m' = 1' to M') acquired when a network scan is executed in a communication environment where the network scan result is unsuccessful due to "network congestion" and the communication characteristic {CMbest m '} _known}_3 (m' = 1' to M') that is most suitable for estimating that the network scan result is unsuccessful due to "network congestion".

Furthermore, the host information database 13 stores the scan result SC1 or the scan result SC2.

The extraction means 14 executes an extraction process to extract a communication characteristic CMbest_unknown that is most suitable for estimating the result of a network scan, using a method described later, based on the communication characteristic CM1 (or the communication characteristic CM2) stored in the host information database 13. Then, the extraction means 14 detects a scan response delay RTT _m (hereinafter, referred to as "scan response delay RTT _m _unknown") from the communication characteristic CM1 (or the communication characteristic CM2), generates sample data D_unknown consisting of the detected scan response delay RTT _m _unknown (m = 1 to M) and the communication characteristic CMbest _m _unknown (m = 1 to M), and outputs the generated sample data D_unknown to the calculation means 15 and the estimation means 16.

In addition, the extraction means 14 detects the scan response delay {RTT _m' _known}_1 (m' = 1' to M') and communication characteristics {CMbest _m ' _known}_1 (m' = 1' to M') acquired in a communication environment where the result of the network scan is "normal" from the correspondence table TBL3 stored in the host information database 13, generates standard sample data D_standard_1 consisting of the detected scan response delay {RTT _m' _known}_1 (m' = 1' to M') and communication characteristics {CMbest _m' _known}_1 (m' = 1' to M'), and outputs the generated standard sample data D_standard_1 to the calculation means 15.

Furthermore, the extraction means 14 detects the scan response delay {RTT _m' _known}_2 (m' = 1' to M') and the communication characteristic {CMbest _m ' _known}_2 (m' = 1' to M') acquired in the communication environment where the result of the network scan is "insufficient communication quality" from the correspondence table TBL3 stored in the host information database 13, and extracts standard sample data D_s consisting of the detected scan response delay {RTT _m' _known}_2 (m' = 1' to M') and the communication characteristic {CMbest _m' _known}_2 (m' = 1' to M').
The standard sample data D_standard_2 is generated, and the generated standard sample data D_standard_2 is output to the calculation means 15.

Furthermore, the extraction means 14 detects the scan response delay {RTT _m' _known}_3 (m' = 1' to M') and communication characteristics {CMbest _m ' _known}_3 (m' = 1' to M') acquired in a communication environment where the result of the network scan is "network congestion" from the correspondence table TBL3 stored in the host information database 13, generates standard sample data D_standard_3 consisting of the detected scan response delay {RTT _m' _known}_3 (m' = 1' to M') and communication characteristics {CMbest _m' _known}_3 (m' = 1' to M'), and outputs the generated standard sample data D_standard_3 to the calculation means 15.

Furthermore, the extraction means 14 receives from the calculation means 15 three similarities P S1 , P S2 , and P S3 between each of the scan response delays {RTT _{m' _known} }_1 (m' = 1' to M') to {RTT _m' _known}_3 (m' = 1' to M' ₎ included in the standard sample data _{D_standard_1} to _{D_standard_3} and the scan response delay RTT m _known included in the sample data D_unknown, and calculates the scan response delay { _{RTT m' _known }} _similarity (m' = 1' to M') (= the scan response delay {RTT _m' _known}_1 (m'=1' to M') to {RTT _m' _known}_3 (m'=1' to M') are detected. Then, extraction means 14 outputs to estimation means 16 standard sample data D_similarity consisting of scan response delay {RTT _m' _known}_similarity (m' = 1' to M') and communication characteristic {CMbest _{m '} _known} similarity (= any of communication characteristics {CMbest m' _known}_1 (m' = 1' to M') to {CMbest _{m '} _known}_3 (m' = 1' to M')) corresponding to the scan response delay {RTT m' _known}_similarity (m' = 1' to M').

The calculation means 15 receives the sample data D_unknown and the standard sample data D_standard_1 to D_standard_3 from the extraction means 14. Then, the calculation means 15 calculates three similarities P S1 , P S2 , and P S3 between each of the scan response delays {RTT _m' _known}_1 (m' = 1' to M') to {RTT m _' _known}_3 (m' = 1' to M') included in the standard sample data D_standard_1 to D_standard_3 and the scan response delay RTT _m _known (m ₌ 1 to M) included in the sample data D_unknown by _{a method described later, and outputs the three calculated similarities P S1 , P S2 , and P S3} to the extraction means 14.

The calculation means 15 also receives from the estimation means 16 the estimation results (confusion matrices CFM, nCFM, tCFM) obtained by estimating the results of the network scan using the estimation algorithm ALG. The calculation means 15 then calculates an evaluation index EVD representing the accuracy of the estimation result based on the confusion matrix CFM using a method described below, calculates an evaluation index nEVD representing the accuracy of the estimation result based on the confusion matrix nCFM, and calculates an evaluation index tEVD representing the accuracy of the estimation result based on the confusion matrix tCFM. The calculation means 15 then outputs the evaluation indexes EVD, nEVD, and tEVD to the estimation means 16.

The estimation means 16 combines the standard sample data D_standard_1 to D_standard_3 to generate initial sample data D_sample_initial for modeling.

The estimation means 16 also receives standard sample data D_similarity (consisting of scan response delay {RTT _m' _known}_similarity (m'=1' to M') and communication characteristic {CMbest _m' _known}_similarity (m'=1' to M')) from the extraction means 14. Then, the estimation means 16 divides the standard sample data D_similarity into training data and test data, and divides the training data into a training set and a validation set. Then, the estimation means 16 generates a plurality of tentative sample data D_sample_tentative based on the training set and initial sample data D_sample_initial by a method described later.

Then, the estimation means 16 trains the estimation algorithm ALG using the initial sample data D_sample_initial, estimates the results of the network scan by the estimation algorithm ALG (the trained estimation algorithm ALG) using the validation set, and generates a confusion matrix CFM based on the estimated results of the network scan and outputs it to the calculation means 15.

The estimation means 16 then trains the estimation algorithm ALG using the tentative sample data D_sample_tentative, estimates the results of the network scan with the estimation algorithm ALG (the trained estimation algorithm ALG) using the validation set, and generates a confusion matrix nCFM based on the estimated results of the network scan and outputs it to the calculation means 15.

The estimation means 16 outputs the confusion matrix CFM to the calculation means 15, and then receives the evaluation index EVD from the calculation means 15, and outputs the confusion matrix nCFM to the calculation means 15, and then receives the evaluation index nEVD from the calculation means 15. Then, when the evaluation index nEVD is greater than the evaluation index EVD, the estimation means 16 sets the evaluation index nEVD to the evaluation index EVD. On the other hand, when the evaluation index nEVD is not greater than the evaluation index EVD, the estimation means 16 does not set the evaluation index nEVD to the evaluation index EVD.

The estimation means 16 performs a comparison between the evaluation index nEVD and the evaluation index EVD for all of the multiple tentative sample data D_sample_tentative, and replaces the variables of the estimation algorithm ALG with the tentative sample data D_sample_tentative when the largest evaluation index nEVD is obtained.

Then, the estimation means 16 estimates the results of the network scan by the estimation algorithm ALG (the estimation algorithm ALG with the variables replaced) using the test data, generates a confusion matrix tCFM based on the estimated results of the network scan, and outputs it to the calculation means 15. Then, the estimation means 16 receives the evaluation index tEVD from the calculation means 15, and when the evaluation index tEVD is equal to or greater than the evaluation index EVD, it approves the estimation algorithm ALG with the variables replaced and the tentative sample data D_sample_tentative when the largest evaluation index nEVD is obtained.

In this way, the estimation means 16 trains the estimation algorithm ALG and evaluates the accuracy of the estimation result when the network scan result is estimated using the trained estimation algorithm ALG.

Furthermore, the estimation means 16 estimates the results of the network scan using the estimation algorithm ALG in the following manner: The estimation means 16 classifies the terminal devices having the communication characteristics CMbest _m _unknown (m=1 to M) extracted by the extraction means 14 into a group CLS ₁ in which the network scan is normal, and groups CLS ₂ and CLS ₃ in which the network scan has failed due to a cause attributable to the communication channel with the terminal device.

The estimation means 16 then calculates a D-score D_2 for distinguishing between groups CLS ₁ and CLS ₃ , or a D-score D_3 for distinguishing between groups CLS ₂ and CLS ₃ , and estimates that the network scan is normal or unsuccessful based on the calculated D-score D_2 or D-score D_3, and when it estimates that the network scan is unsuccessful, executes an estimation process for estimating the cause of the failure of the network scan. The estimation means 16 then outputs the estimation result estimated in the estimation process to the scan scheduler 17.

The scan scheduler 17 reads the scan results from the host information database 13 and receives scan requirements from the user 30. The scan scheduler 17 also holds in advance the global IP addresses and domain names of all terminal devices present in the area (e.g., Japan) where the network scan is to be performed.

Then, based on the scan requirements and the scan results, the scan scheduler 17 generates a scan schedule SCH_scan that associates the scan timings of the network scan with the terminal devices that are the targets of the network scan at each scan timing, using a method described below. The scan scheduler 17 then outputs the scan schedule SCH_scan to the network scanner 11 and controls the network scanner 11 to execute a network scan in accordance with the scan schedule SCH_scan.

In addition, when the scan scheduler 17 receives the estimation result from the estimation means 16, it reschedules the scan schedule based on the cause of the network scan failure in the estimation result, outputs the rescheduled scan schedule RESCH_scan to the network scanner 11, and controls the network scanner 11 to execute the network scan in accordance with the scan schedule RESCH_scan.

Figure 3 is a conceptual diagram of the correspondence table TBL1 stored in the host information database 13 shown in Figure 2. Note that the correspondence table TBL1 shown in Figure 3 is a correspondence table when a network scan is performed without monitoring background traffic.

Referring to FIG. 3, correspondence table TBL1 includes time t, an IP address Add _{b_v} (v=1 to V, where V is the total number of base stations), an IP address Add _m , a domain name DN _m , a port number PN _r (r is a positive integer), a scan response SR _m , a scan response delay RTT _m , a packet loss rate PLR _m , a scan maximum delay DLmax _m , and a scan minimum delay DLmin _m .

Time t is expressed as YYYY/MM/DD/hh/mm/ss (year/month/day/hour/minute/second). Time t is a time within one scan timing having a certain time length, which will be described later, and represents the time when the test input in the network scan is sent to the terminal device that is the target of the network scan.

The IP address Add _{b_v} is the address of the base station, and the IP address Add _m is the global IP address of the terminal device that is the target of the network scan.

The domain name DN _m is converted based on the IP address Add _m using a PTR (Pointer record).

The port number _PNr is the number of the port that each terminal device uses for communication.

The scan response _SRm is set to "1" when a scan response is returned, and is set to "0" when no scan response is returned.

The scan response delay RTT _m is the time from the time the test input is transmitted until the scan response SR _m is returned.

The packet loss rate PLR _m is the rate at which packets fail to be transmitted and received when a packet consisting of a test input is transmitted and a scan response SR _m is received.

The scan maximum delay DLmax _m is the maximum value of the scan response delay RTT _m , and the scan minimum delay DLmin _m is the minimum value of the scan response delay RTT _m .

The scan response SR _m and the scan response delay RTT _m are associated with the port number PN _r . The packet loss rate PLR _m , the scan maximum delay DLmax _m and the scan minimum delay DLmin _m are associated with the IP address Add _m of the terminal device that is the target of the network scan.

Furthermore, addresses Add ₁ to Add ₃ are associated with address Add _{b_1} , ..., addresses Add _M-1 to Add _M are associated with address Add _{b_V} . As a result, it is possible to know the terminal devices belonging to each base station.

Figure 4 is a conceptual diagram of another correspondence table TBL2 stored in the host information database 13 shown in Figure 2. Note that the correspondence table TBL2 shown in Figure 4 is a correspondence table used when a network scan is performed by monitoring background traffic.

4, correspondence table TBL2 is the same as correspondence table TBL1, except that the scan maximum delay DLmax _m and scan minimum delay DLmin _m of correspondence table TBL1 shown in FIG. 3 are replaced with delay DL _m , jitter JT _m , background maximum delay BGDLmax _m and background minimum delay BGDLmin _m .

The delay DL _m , the jitter JT _m , the background maximum delay BGDLmax _m and the background minimum delay BGDLmin _m are associated with the IP address Add _m of each terminal device.

The delay DL _m is the delay from the standard time in one-way communication from the terminal device.

Jitter JT _m is the fluctuation in the transmission time of a packet in one-way communication from a terminal device.

The background maximum delay BGDLmax _m is the maximum value of the transmission delay in one-way communication from the terminal device, and the background minimum delay BGDLmin _m is the minimum value of the transmission delay in one-way communication from the terminal device.

In the correspondence table TBL2, the scan response SR _m and the scan response delay RTT _m are associated with the port number PN _r of each terminal device.

Figure 5 is a conceptual diagram of yet another correspondence table TBL3 stored in the host information database 13 shown in Figure 3.

5, correspondence table TBL3 includes a known estimation result, a time t', an IP address Add _{b'_w} (w=1 to W, where W is the total number of base stations), an IP address Add _m' , a scan response delay RTT _m' _known, and a communication characteristic CMbest _m' _known. The known estimation result, the time t', the IP address Add _{b'_w} , the IP address Add _m' , the scan response delay RTT _m' _known, and the communication characteristic CMbest _m' _known are associated with each other.

Known estimated results from network scans include "normal," "poor communication quality," and "network congestion."

The time t' has the same format as the time t described in Fig. 3. The IP address Add _{b'_w} is the address of the base station. The IP address Add _m' is the global IP address of the terminal device that is the target of the network scan. The IP addresses Add _1' to Add _3' are associated with the IP address Add _{b'_1} of the base station, and similarly, the IP address Add _M' is associated with the IP address Add _{b'_W} of the base station.

The scan response delay {RTT _1' _known}_1, the scan response delay {RTT _2' _known}_1, the scan response delay {RTT _3' _known}_1, ..., and the scan response delay {RTT _M' _known}_1 are respectively associated with IP address Add _1' , IP address Add _2' , IP address Add _3' , ..., IP address Add _M' , and are scan response delays RTTs obtained in a communication environment where the result of a network scan is "normal".

Communication characteristic {CMbest _1' _known}_1, communication characteristic {CMbest _2' _known}_1, communication characteristic {CMbest _3' _known}_1, ..., and communication characteristic {CMbest _M' _known}_1 are communication characteristics CMbest respectively associated with IP address Add _1' , IP address Add _2' , IP address Add _3' , ..., IP address Add _M' , and are obtained in a communication environment where the result of a network scan is "normal".

The scan response delay {RTT _1' _known}_2, the scan response delay {RTT _2' _known}_2, the scan response delay {RTT _3' _known}_2, ..., and the scan response delay {RTT _M' _known}_2 are respectively associated with IP address Add _1' , IP address Add _2' , IP address Add _3' , ..., IP address Add _M' , and are scan response delays RTTs obtained in a communication environment where the result of a network scan is "insufficient communication quality."

The communication characteristic {CMbest _1' _known}_2, the communication characteristic {CMbest _2' _known}_2, the communication characteristic {CMbest _3' _known}_2, ..., and the communication characteristic {CMbest _M' _known}_2 are communication characteristics CMbest that are associated with IP address Add _1' , IP address Add _2' , IP address Add _3' , ..., IP address Add _M' , respectively, and are acquired in a communication environment where the result of a network scan is "insufficient communication quality."

The scan response delay {RTT _1' _known}_3, the scan response delay {RTT _2' _known}_3, the scan response delay {RTT _3' _known}_3, ..., and the scan response delay {RTT _M' _known}_3 are scan response delays RTTs associated with IP address Add _1' , IP address Add _2' , IP address Add ₃ ', ..., IP address Add _M' , respectively, and obtained in a communication environment where the result of a network scan is "network congestion."

The communication characteristic {CMbest _1' _known}_3, the communication characteristic {CMbest _2' _known}_3, the communication characteristic {CMbest _3' _known}_3, ..., and the communication characteristic {CMbest _M' _known}_3 are communication characteristics CMbest that are associated with IP address Add _1' , IP address Add _2' , IP address Add _3' , ..., IP address Add _M' , respectively, and are acquired in a communication environment where the result of a network scan is "network congestion".

The scan response delays {RTT _{1 '} _known}_1 to {RTT _{M'_known} }_1 and the communication characteristics {CMbest _{1 '} _known}_1 to {CMbest _{M'_known} }_1 constitute standard sample data D_standard_1.

Furthermore, the scan response delays {RTT _{1 '} _known}_2 to {RTT _{M'_known} }_2 and the communication characteristics {CMbest _{1 '} _known}_2 to {CMbest _{M'_known} }_2 constitute standard sample data D_standard_2.

Furthermore, the scan response delays {RTT _{1 '} _known}_3 to {RTT _{M'_known} }_3 and the communication characteristics {CMbest _{1 '} _known}_3 to {CMbest _{M'_known} }_3 constitute standard sample data D_standard_3.

In addition, each of the scan response delays {RTT _1' _known}_1 to {RTT _M' _known}_1 shown in the correspondence table TBL3 actually exists in a plurality of pieces corresponding to a plurality of ports in one IP address Add _m' . Also, each of the communication characteristics {CMbest _1' _known}_1 to {CMbest _M' _known}_1 actually exists in a plurality of pieces corresponding to a plurality of ports in one IP address Add _m' . The same is true for the scan response delays {RTT _{1 '} _known}_2 to {RTT _{M'_known} }_2; {RTT _{1 '} _known}_3 to {RTT _{M'_known} }_3 and the communication characteristics {CMbest _{1 '} _known}_2 to {CMbest _{M'_known} }_2; {CMbest _{1 '} _known}_3 to {CMbest _{M'_known} }_3.

Further, the “1” of the scan response delays {RTT _1′ _known}_1 to {RTT _M′ _known}_1 and the “1” of the communication characteristics {CMbest _1′ _known}_1 to {CMbest _M′ _known}_1 represent an identification code D_code=1 assigned to “normal”, which is a known estimated result of the network scan. The “2” of the scan response delays {RTT _1′ _known}_2 to {RTT _M′ _known}_2 and the “2” of the communication characteristics {CMbest _1′ _known}_2 to {CMbest _M′ _known}_2 represent an identification code D_code=2 assigned to “insufficient communication quality”, which is a known estimated result of the network scan. The "3" in {RTT _{1'_known} }_3 to {RTT _{M'_known} }_3 and the "3" in {CMbest _{1'_known} }_3 to {CMbest _{M'_known} }_3 represent the identification code D_code=3 assigned to "network is congested", which is a known estimated result of the network scan.

In this embodiment of the invention, when a network scan is performed without monitoring background traffic, correspondence tables TBL1 and TBL3 are stored in the host information database 13, and when a network scan is performed while monitoring background traffic, correspondence tables TBL2 and TBL3 are stored in the host information database 13. Correspondence table TBL3 is stored in the host information database 13 before the network scanner 11 starts a network scan in a communication environment where the results of the network scan are unknown.

Fig. 6 is a conceptual diagram of a scan schedule. Referring to Fig. 6(a), the scan schedule SCH_scan has a total scan time length T _S _total, and includes T (T is an integer equal to or greater than 1) scan timings 1 _S to T _S.

The total scan time length T _S _total is set to, for example, one day, and each of scan timings 1 _S to _TS has a time length of, for example, one hour. The scan timings t _S (t _S =1 _S to _TS ) include a start time t _start , an end time t _end , and target addresses Add ₁ to Add _M.

The start time t _start represents the time when the network scan starts at the scan timing t _S , and the end time t _end represents the time when the network scan ends at the scan timing t _S. Therefore, the network scanner 11 ends the network scan when the end time t _end arrives at the scan timing t _S.

Each of the target addresses _{Add 1} to Add _M is a global IP address, and represents _M terminal devices that are targets of the network scan.

Each of the scan timings 1 _S to (t-1) _S and (t+1) _S to T _S other than the scan timing t _S has the same configuration as the scan timing t _S shown in FIG. 6( a ).

The scan timings _1S to _TS may have the same time length (=t _end -t _start ) or may have different time lengths (=t _end -t _start ). Also, the scan timings _1S to _TS may have the same number of IP addresses or may have different numbers of IP addresses.

When a network scan is executed for the first time, the scan timing 1 _S to T _S shown in FIG. 6(a) is used.

Furthermore, when a network scan is performed during times when communication is not busy, the scan timings 1 _'S to T _'S (T' is an integer equal to or greater than 1) shown in Fig. 6(b) are used. In Fig. 6(b), the scan timings 1 _'S to T _'S are set to be spaced apart in time, because multiple time periods during which communication is not busy are never adjacent to each other.

The configuration of the scan timing t' _S (t' _S =1' _S to T' _S ) is the same as the configuration of the scan timing t _S described above.

Moreover, the scan timings t _S and t' _S do not mean a single time, but are a concept having a predetermined time length from the start time t _start to the end time t _end .

A method for determining the scan timing t _'S will be described. Fig. 7 is a diagram showing the time dependency of traffic. In Fig. 7, the vertical axis represents traffic and the horizontal axis represents time. Furthermore, curve k1 represents the time dependency of traffic in, for example, terminal device A, and curve k2 represents the time dependency of traffic in, for example, terminal device B different from terminal device A.

Referring to (a) of FIG. 7, in terminal device A, traffic is heavier at night than during the day (see curve k1). That is, in terminal device A, communication is less busy during the day and congested at night. Referring to (b) of FIG. 7, in terminal device B, traffic is heavier during the day than at night (see curve k2). That is, in terminal device B, communication is congested during the day and less busy at night.

Therefore, the scan scheduler 17 detects a time period during which communication is not busy for the M terminal devices that are the targets of the network scan, and determines the scan timing t _S (or the scan timing t' _S ) based on the detected time period.

The network scanner 11 receives the scan schedule SCH_scan from the scan scheduler 17 and performs a network scan on M terminal devices in accordance with the received scan schedule SCH_scan.

In this case, network scanning is performed without monitoring background traffic, or with monitoring background traffic.

Fig. 8 is a diagram showing correlation coefficients of communication characteristics. The correlation coefficients shown in Fig. 8 are correlation coefficients of communication characteristics when communication quality is insufficient when a network scan is performed without monitoring background traffic. The scan response delay RTT _m , packet loss rate PLR _m , scan maximum delay DLmax _m , and scan minimum delay DLmin _m shown in Fig. 8 are multiple communication characteristics of M terminal devices when a network scan is performed on M terminal devices. The scan response delay RTT _m indicates one scan response delay or an average value of multiple scan response delays within one IP address that is the target of a network scan.

The correlation coefficients between the scan response delay RTT _m , the packet loss rate PLR _m , the scan maximum delay DLmax _m , and the scan minimum delay DLmin _m are shown in Fig. 8. Note that in Fig. 8, the correlation coefficients of the off-diagonal elements above the diagonal elements are not shown because they are the same as the correlation coefficients of the off-diagonal elements below the diagonal elements.

8, when a network scan is performed without monitoring background traffic and communication quality is insufficient, the correlation coefficient between the scan response delay RTT _m and the scan maximum delay DLmax _m , the correlation coefficient between the scan response delay RTT _m and the scan minimum delay DLmin _m , and the correlation coefficient between the scan maximum delay DLmax _m and the scan minimum delay DLmin _m are greater than 0.5. The correlation coefficient between the scan response delay RTT _m and the scan maximum delay DLmax _m and the correlation coefficient between the scan response delay RTT _m and the scan minimum delay DLmin _m are greater than the correlation coefficient between the scan maximum delay DLmax _m and the scan minimum delay DLmin _m . Therefore, the scan response delay RTT _m has a large correlation with the scan maximum delay DLmax _m and the scan minimum delay DLmin _m .

FIG. 9 is a diagram showing another correlation coefficient of communication characteristics. The correlation coefficient shown in FIG. 9 is a correlation coefficient of communication characteristics when a network scan is performed without monitoring background traffic and the network is congested (network congestion). The scan response delay RTT _m indicates an average value within one IP address that is a target of a network scan. The mutual correlation coefficients of the scan response delay RTT _m , the packet loss rate PLR _m , the scan maximum delay DLmax _m , and the scan minimum delay DLmin _m are shown in FIG. 9. The reason why the correlation coefficients of the off-diagonal elements above the diagonal elements are not shown in FIG. 9 is the same as that explained in FIG. 8. The scan response delay RTT _m , the packet loss rate PLR _m , the scan maximum delay DLmax _m , and the scan minimum delay DLmin _m shown in FIG. 9 are a plurality of communication characteristics of M terminal devices when a network scan is performed on M terminal devices.

9, in the case where a network scan is performed without monitoring background traffic, when the network is congested, the correlation coefficient between the scan response delay RTT _m and the scan maximum delay DLmax _m , the correlation coefficient between the scan response delay RTT _m and the scan minimum delay DLmin _m , and the correlation coefficient between the scan maximum delay DLmax _m and the scan minimum delay DLmin _m are greater than 0.5. The correlation coefficient between the scan response delay RTT _m and the scan maximum delay DLmax _m and the correlation coefficient between the scan response delay RTT _m and the scan minimum delay DLmin _m are the same. Also, the correlation coefficient between the scan response delay RTT _m and the scan maximum delay DLmax _m and the correlation coefficient between the scan response delay RTT _m and the scan minimum delay DLmin _m are greater than the correlation coefficient between the scan maximum delay DLmax _m and the scan minimum delay DLmin _m . Therefore, even when the network is congested, the scan response delay RTT _m has a large correlation with the scan maximum delay DLmax _m and the scan minimum delay DLmin _m .

As shown in Figures 8 and 9 , when a network scan is performed without monitoring background traffic, the correlation coefficients between the scan response delay RTT _m , packet loss rate PLR _m , scan maximum delay DLmax _m , and scan minimum delay DLmin _m are different when the communication quality is insufficient and when the network is congested.

Figure 10 is a diagram showing yet another correlation coefficient of communication characteristics. Note that the correlation coefficient shown in Figure 10 is the correlation coefficient of communication characteristics when communication quality is insufficient when a network scan is performed by monitoring background traffic.

The correlation coefficients between the scan response delay RTT _m , packet loss rate PLR _m , delay DL _m , jitter JT _m , background maximum delay BGDLmax _m , and background minimum delay BGDLmin _m are shown in Fig. 10. Note that the correlation coefficients of the off-diagonal elements above the diagonal elements are not shown in Fig. 10 for the same reason as explained in Fig. 8. The scan response delay RTT _m , packet loss rate PLR _m , delay DL _m , jitter JT _m , background maximum delay BGDLmax _m , and background minimum delay BGDLmin _m shown in Fig. 10 are multiple communication characteristics of M terminal devices when a network scan is performed on M terminal devices.

Referring to FIG. 10 , when a network scan is performed by monitoring background traffic and the communication quality is insufficient, the correlation coefficient between the scan response delay RTT _m and the packet loss rate PLR _m , the delay DL _m , the background maximum delay BGDLmax _m , and the background minimum delay BGDLmin _m , the correlation coefficient between the packet loss rate PLR _m and the delay DL _m , the background maximum delay BGDLmax _m , and the background minimum delay BGDLmin _m , the correlation coefficient between the delay DL _m and the jitter JT _m , the background maximum delay BGDLmax _m , and the background minimum delay BGDLmin _m , the correlation coefficient between the jitter JT _m and the background maximum delay BGDLmax _m and the background minimum delay BGDLmin _m , and the correlation coefficient between the background maximum delay BGDLmax _m and the background minimum delay BGDLmin _m are greater than 0.5. The scan response delay RTT _m has the largest correlation coefficient with the packet loss rate PLR _m .

Figure 11 is a diagram showing yet another correlation coefficient of communication characteristics. Note that the correlation coefficient shown in Figure 11 is the correlation coefficient of communication characteristics when the network is congested when a network scan is performed by monitoring background traffic.

The correlation coefficients between the scan response delay RTT _m , packet loss rate PLR _m , delay DL _m , jitter JT _m , background maximum delay BGDLmax _m , and background minimum delay BGDLmin _m are shown in Fig. 11. Note that the correlation coefficients of the off-diagonal elements above the diagonal elements are not shown in Fig. 11 for the same reason as explained in Fig. 8. The scan response delay RTT _m , packet loss rate PLR _m , delay DL _m , jitter JT _m , background maximum delay BGDLmax _m , and background minimum delay BGDLmin _m shown in Fig. 11 are a plurality of communication characteristics of M terminal devices when a network scan is performed on the M terminal devices.

Referring to FIG. 11 , when a network scan is performed by monitoring background traffic, when the network is congested, the correlation coefficient between the scan response delay RTT _m and the packet loss rate PLR _m , the delay DL _m , the jitter JT _m , and the background maximum delay BGDLmax _m , the correlation coefficient between the packet loss rate PLR _m and the delay DL _m , the background maximum delay BGDLmax _m , and the background minimum delay BGDLmin _m , the correlation coefficient between the delay _{DL m} and the jitter JT _m , the background maximum delay BGDLmax _m , and the background minimum delay BGDLmin _m , the correlation coefficient between the jitter JT _m and the background maximum delay BGDLmax _m and the background minimum delay BGDLmin m, and the correlation coefficient between the background maximum delay BGDLmax _m and the background minimum delay BGDLmin _m are greater than 0.5. And the scan response delay RTT _m has the largest correlation coefficient with the background maximum delay BGDLmax _m .

As shown in Figures 10 and 11, when a network scan is performed by monitoring background traffic, the correlation coefficients between the scan response delay RTT _m , packet loss rate PLR _m , delay DL _m , jitter JT _m , background maximum delay BGDLmax _m , and background minimum delay BGDLmin _m differ when communication quality is insufficient and when the network is congested.

8 to 11 may be correlation coefficients calculated by using the scan response delay RTT _m stored in the correspondence table _TBL1 shown in _{FIG .} 3 or the correspondence table TBL2 shown in FIG. 4 as the scan response _delay RTT _m , which is averaged by the number of ports at _{which a scan response SR m was} received in one terminal device, or may be correlation coefficients calculated by using each communication characteristic such as the scan response delay RTT _m and the packet loss rate PLR _m stored in the correspondence tables TBL1 and TBL2 _{as they are (i.e., with regard to the scan response delay RTT m ,} taking into consideration differences in the communication characteristics of the scan response delay RTT _m due to differences in ports).

When a network scan is performed, the results of the network scan are classified as successful or failed. If the network scan fails, the following first, second, third, and fourth causes are assumed to be the reasons why the network scan failed.

(First cause)
The communication quality with the terminal device m that is the target of the network scan is insufficient.

(Second cause)
The network to which the terminal device m that is the target of the network scan is connected is congested (i.e., network congestion).

(Third cause)
The terminal device m that is the target of the network scan is not connected to the network.

(Fourth cause)
The port of the terminal device m that is the target of the network scan is blocked.

Therefore, in the embodiment of the present invention, the communication characteristics such as the scan response delay RTT _m , packet loss rate PLR _m , and delay DL _m obtained when a network scan is executed are used to estimate whether the network scan is normal or unsuccessful, and when it is estimated that the network scan has failed, the cause of the failure (any of the first, second, third, or fourth cause) is estimated. Note that the first and second causes are causes attributable to the communication channel with the terminal device.

The method for estimating the result of a network scan includes an extraction process for extracting a communication characteristic CMbest that is most suitable for estimating the result of a network scan from communication characteristics such as a scan response delay RTT _m , a packet loss rate PLR _m , and a delay DL _m shown in Figures 8 to 11, and an estimation process for classifying terminal devices having the communication characteristics extracted in the extraction process into a group CLS ₁ where the network scan is normal and groups CLS ₂ and CLS ₃ where the network scan has failed due to a cause caused by _the communication channel with the terminal _device , calculating a D-score D_2 for distinguishing between groups CLS ₁ and CLS ₃ , or a D-score D_3 for distinguishing between groups CLS 2 and CLS 3, estimating that the network scan is normal or unsuccessful based on the calculated D-score D_2 or D-score D_3, and estimating the cause of the failure of the network scan when it is estimated that the network scan has failed.

[Extraction of communication characteristic CMbest _m _unknown]
When a network scan is performed without monitoring background traffic according to the schedule SCH_scan shown in FIG. 6 , the scan analysis manager 12 receives the scan response delay RTT _m , packet loss rate PLR _m , scan maximum delay DLmax _m , and scan minimum delay DLmin _m at one scan timing _ts for each of the M terminal devices from the network scanner 11, and creates or updates the host information database 13 by storing the received scan response delay RTT _m , packet loss rate PLR _m , scan maximum delay DLmax _m , and scan minimum delay DLmin _m in the correspondence table TBL1 of the host information database 13.

The extraction means 14 reads out M scan response delays RTT _m , M packet loss rates PLR _m , M maximum scan delays DLmax _m and M minimum scan delays DLmin _m from the correspondence table TBL1 of the host information database 13, and calculates the mutual correlation coefficient r ij of the scan response delays RTT _m , packet loss rates PLR _m , M maximum scan delays DLmax _m and M minimum scan delays DLmin _m based on the read out M scan response delays RTT _m , M packet loss rates PLR _m , M maximum scan delays DLmax _m and M minimum scan delays DLmin _m using the following _formula :

In formula (1), x _ai represents a sample of one communication characteristic f _i from which the correlation coefficient r _ij is calculated, and y _aj represents a sample of the other communication characteristic f _j from which the correlation coefficient r _ij is calculated, where a _i =1 to M and a _j =1 to M.

The extraction means 14 extracts correlation coefficients r _ij (i≠j) of different communication characteristics f _i , f _j (i≠j) from among the correlation coefficients r _ij of the communication characteristics f _i , f _j . For example, when the calculated correlation coefficients r _ij are those shown in Fig. 8, the communication characteristics f _i (i=1 to 4) are set to the scan response delay RTT _m , packet loss rate PLR _m , scan maximum delay DLmax _m and scan minimum delay DLmin _m in the column direction, respectively, and the communication characteristics f _j (j=1 to 4) are set to the scan response delay RTT _m , packet loss rate PLR _m , scan maximum delay DLmax _m and scan minimum delay DLmin _m in the row direction, respectively.

The correlation coefficients _r11 , _r22 , _r33 , and _r44 are diagonal elements, and the correlation coefficients _r21 , _r31 , _r41 , _r32 , _r42 , and _r43 are off-diagonal elements.

The extraction means 14 sequentially extracts correlation coefficients _r21 , r31, _r41 , _r32 , _r42 , and _r43 of the off-diagonal elements, and judges whether each of the extracted correlation coefficients _r21 , _r31 , _r41 , _r32 , _r42 , and _r43 is greater than a threshold value Corr _r . _The threshold value Corr _r is set to a real number in the range of 0.5 to 1.0, for example. Here, the threshold value Corr _r = 0.5.

First, the extraction means 14 judges whether the correlation coefficient _r21 is greater than a threshold value _{Corr_r} . If the correlation coefficient _r21 is equal to or less than the threshold value _{Corr_r} , the extraction means 14 deletes the two communication characteristics _f2 and _f1 used to calculate the correlation coefficient _r21 , and if the correlation coefficient _r21 is greater than the threshold value _{Corr_r} , the extraction means 14 stores the two communication characteristics _f2 and _f1 used to calculate the correlation coefficient _r21 .

In this case, since the correlation coefficient _r21 (=0.25) is less than the threshold value Corr _r (=0.5), the extraction means 14 deletes the two communication characteristics _f2 (=packet loss rate _PLRm ) and communication characteristic _f1 (=scan response delay _RTTm ) that were used to calculate the correlation coefficient _r21 .

Next, the extraction means 14 judges whether the correlation coefficient _r31 is greater than the threshold value _Corr3r . Since the correlation coefficient _r31 is 0.80, the extraction means 14 judges that the correlation coefficient _r31 is greater than the threshold value _Corr3r , and saves the two communication characteristics _f3 (=scan maximum delay _DLmaxm ) and _f1 (=scan response delay _RTTm ) that were used to calculate the correlation coefficient _r31 .

Next, the extraction means 14 judges whether the correlation coefficient _r41 is greater than the threshold value _Corr41 . Since the correlation coefficient _r41 is 0.97, the extraction means 14 judges that the correlation coefficient _r41 is greater than the threshold value _Corr41 , and stores the two communication characteristics _f3 (=scan minimum delay _DLminm ) and _f1 (=scan response delay _RTTm ) that were used to calculate the correlation coefficient _r41 .

Similarly, the extraction means 14 sequentially determines whether the correlation coefficients _r32 , _r42 , and _r43 are greater than the threshold value _Corrr . The extraction means 14 then stores the two communication characteristics _f4 (=minimum scanning delay _DLminm ) and _f3 (=maximum scanning delay DLmaxm) that were used to calculate the correlation coefficient _r43 that is greater than the threshold value _Corrr (=0.5). The extraction means 14 also deletes the two communication characteristics _f3 (=maximum scanning delay _DLmaxm ) and _f2 (=packet loss rate _PLRm ) that were used to calculate the correlation coefficients _r32 and _r42 that are less than the threshold value Corrr (= _0.5 ); _and the two communication characteristics _f4 (=minimum scanning delay _DLminm ) and _f2 (=packet loss rate _PLRm ).

As a result, two communication characteristics _f3 (= maximum scan delay DLmax _m ), _f1 (= scan response delay RTT _m ); two communication characteristics _f3 (= minimum scan delay DLmin _m ), _f1 (= scan response delay RTT _m ); and two communication characteristics _f4 (= minimum scan delay DLmin _m ), _f3 (= maximum scan delay DLmax _m ) are saved, so the extraction means 14 finally saves three communication characteristics: scan response delay RTT _m , maximum scan delay DLmax _m and minimum scan delay DLmin _m .

In addition, when the correlation coefficient r _ij is the correlation coefficient shown in any one of Figures 9 to 11, the extraction means 14 similarly stores the communication characteristics used to calculate the correlation coefficient r _ij that is greater than the threshold value Corr _r , and deletes the communication characteristics used to calculate the correlation coefficient r _ij that is equal to or smaller than the threshold value Corr _r .

The scan response delay RTT _m represents the delay from sending a packet of test input to receiving a scan response in a normal network scan, relative to the RTT _normal , and therefore best represents the results of a network scan.

Therefore, the extraction means 14 extracts the maximum scan delay DLmax _m and the minimum scan delay DLmin _m as explanatory variables from the finally saved communication characteristics (in the above example, scan response delay RTT _m , maximum scan delay DLmax _m , and minimum scan delay DLmin _m ), and extracts the scan response delay RTT _m as a response variable, and performs multiple regression analysis to find the contribution rate (= ^R2 ) in the multiple regression analysis.

Thereafter, the extraction means 14 judges whether the contribution rate ^R2 is smaller than a threshold value _Sr. The threshold value _Sr is set to a real number in the range of 0.5 to 1.0, for example.

When the extraction means 14 determines that the contribution rate ^R2 is smaller than the threshold value _Sr , it sets the mode of the estimation algorithm ALG that estimates the results of the network scan to "uncorrelated," and when it determines that the contribution rate ^R2 is not smaller than the threshold value _Sr , it sets the mode of the estimation algorithm ALG to "correlated."

When the mode of the estimation algorithm ALG is set to "uncorrelated", it means that the communication characteristics finally saved are somewhat important. On the other hand, when the mode of the estimation algorithm ALG is set to "correlated", it means that the communication characteristics finally saved are very important.

The extraction means 14 then uses the machine learning algorithm "SelectKbest" to select the communication characteristic CMbest that is most suitable for estimating the results of the network scan from the K communication characteristics.

A method for selecting the communication characteristic CMbest will be described. FIG. 12 is a diagram showing an example of the original data when selecting the communication characteristic CMbest. FIG. 13 is a diagram for explaining the method for selecting the communication characteristic CMbest.

As multiple communication characteristics obtained from the scan response of the network scan, the packet loss rate PLR _m , jitter JT _m , scan maximum delay DLmax _m , scan minimum delay DLmin _m , and delay DL _m are shown in Fig. 12. In Fig. 12, the packet loss rate PLR _m , jitter JT _m , scan maximum delay DLmax _m , scan minimum delay DLmin _m , and delay DL _m are shown for eight terminal devices.

Then, the extraction means 14 selects, by the _above -mentioned method, for example, a packet loss rate PLR _m , a jitter JT _m , a scanning maximum delay DLmax _m , _and a scanning minimum delay DLmin _m as shown in FIG. 13 as communication characteristics having a correlation coefficient r ij greater than the threshold value Corr r.

Then, the extraction means 14 calculates the sum of each row and the sum of each column in FIG. 13, and calculates a grand total, which is the sum of the sum of each row and the sum of each column.

Next, the extraction means 14 calculates the expected value E _rc of row r and column c for all rows and all columns by the following formula:

In formula (2), e _r represents the sum of the rth row, e _c represents the sum of the cth column, and N _grand represents the grand total.

Fig. 14 is a diagram for explaining a method for calculating a score of a communication characteristic. Fig. 15 is a diagram showing the score of a communication characteristic. Note that Fig. 14 explains a method for calculating a score Q of a maximum scan delay DLmax _m .

14, the leftmost column is the sum and grand total of the rightmost column of FIG. 13, and the second leftmost column is a sample of the scan maximum delay DLmax _m shown in FIG.

The extraction means 14 sequentially substitutes the sum of each row shown in FIG. 14 (=0.049, 0.034, 0.023, 0.038, 0.022, 0.022, 0.021, 0.023) into e _r of formula (2), substitutes the sum of the c-th column shown in FIG. 14 (=0.232) into e _c of formula (2), and substitutes the grand total of FIG. 14 into N _grand of formula (2) to calculate the expected value E _rc .

Then, the extraction means 14 calculates the score Q using the following formula:

In equation (3), O is each value of the maximum scan delay DLmax _m shown in FIG. 14, and E is the expected value E _rc calculated by equation (2).

The score Q represents the magnitude of deviation of each sample O from the expected value E in one communication characteristic (e.g., maximum scan delay DLmax _m ). In formula (3), when each sample O is significantly deviated from the expected value E toward the larger side or when each sample O is significantly deviated from the expected value E toward the smaller side, the score Q is large, and when each sample O is slightly deviated from the expected value E toward the larger side or when each sample O is slightly deviated from the expected value E toward the smaller side. Thus, the score Q represents the characteristics of one communication characteristic (e.g., maximum scan delay DLmax _m ).

After calculating the expected value E, the extraction means 14 calculates O−E, then calculates (O−E) ² , and divides the calculated (O−E) ² by the expected value E to calculate (O−E) ² /E.

Then, the extraction unit 14 calculates the sum of (OE) ² /E for each row as the score Q of the scan maximum delay DLmax _m . That is, the extraction unit 14 calculates the score Q of the scan maximum delay DLmax _m by the formula (3).

The extraction means 14 also calculates the score Q by the above-mentioned method for the packet loss rate PLR _m , jitter JT _m , and scan minimum delay DLmin _m shown in Fig. 13. As a result, the score Q for the packet loss rate PLR _m , jitter JT _m , scan maximum delay DLmax _m , and scan minimum delay DLmin _m is obtained as shown in Fig. 15.

Then, the extraction means 14 selects K (K is a positive integer) communication characteristics in descending order of score Q from the scores Q of the packet loss rate PLR _m , jitter JT _m , scan maximum delay DLmax _m , and scan minimum delay DLmin _m .

For example, when K=3, in the example shown in FIG. 15, the extraction means 14 selects a jitter JT _m , a scan minimum delay DLmin _m and a scan maximum delay DLmax _m .

After selecting K communication characteristics, the extraction means 14 selects a communication characteristic CMbest from the K communication characteristics by the following method.

First, the extraction means 14 classifies the K communication characteristics using classification algorithms such as LR (Linear Regression) and SVM (Support Vector Machine) into a group GP1 of communication characteristics that are important for estimating the results of a network scan and a group GP2 of communication characteristics that are not important for estimating the results of a network scan.

Here, we use a support vector machine (SVM-train) as the classification algorithm.

The SVM is one of the pattern recognition models that uses supervised learning and learns the boundary that most boldly divides the training data X ₀ = [x ₀₁ , x ₀₂ , ..., x _0M ] ^T (T represents transposition). In order to learn this boundary, a two- or more-dimensional space composed of two or more parameters is set.

As a result of learning using learning samples arranged in a two- or more-dimensional space, the boundary that most boldly divides the teacher data _X0 = [ _x01 , _x02 , ..., _x0M ] ^T is determined so as to maximize the distance (margin) between the boundary and the sample (data) closest to the boundary. This determined boundary is called a maximum margin hyperplane. Note that a specific method for determining this boundary is well known and will not be described here.

The extraction means 14 executes learning of a support vector machine (SVM-train) using teacher data X ₀ = [x ₀₁ , x ₀₂ , ..., x _0M ] ^T (T represents transposition), and classifies each of M sample vectors x _m (m = 1 to M) into one of groups GP1 and GP2 using the boundaries determined by the learning, so that each of the M sample vectors x _m (m = 1 to M) satisfies the following formula. Here, the sample vector x _m is x _m = [CM _1m , CM _2m , ..., CM _km , ..., CM _Km ]. That is, the sample vector x _m includes K communication characteristics CM _1m to CM _Km as elements. As a result, the data to be classified into one of groups GP1 and GP2 is expressed by formula (5).

In formula (5), communication characteristic CM _km (k=1 to K, m=1 to M) represents the k-th communication characteristic of terminal device m.

In equation (4), z is a weight vector and b is a predetermined constant.

Group GP1 is a group consisting of important communication characteristics for estimating the results of a network scan. Group GP2 is a group consisting of unimportant communication characteristics for estimating the results of a network scan. When sample vector x _m is classified into group GP1, y _m is "1", and when sample vector x _m is classified into group GP2, y _m is "-1".

The extraction means 14 calculates the weight vector z using equation (4) to determine the classification index α in the following equation.

In formula (6), X = [ _x1 , _x2 , ..., _xM ] ^T (T represents transposition), and y = [ _y1 , _y2 , ..., _yM ] ^T (T represents transposition).

When sample vector _xm is classified into group GP1 (i.e., when _ym is "1"), extraction means 14 calculates weight vector z by z· _xm -b=1 and sets the calculated weight vector z as classification index α. When sample vector _xm is classified into group GP2 (i.e., when _ym is "-1"), extraction means 14 calculates weight vector z by z· _xm -b=-1 and sets the calculated weight vector z as classification index α.

When the extraction means 14 determines the classification index α for the kth communication characteristic CM _km , it substitutes the determined classification index α (= α _k ), the sample vector x _km , and y _m (= “1” or “−1”) into the following equation to calculate a weight w _k indicating the importance of the communication characteristic for estimating the results of the network scan.

In formula (7), x _km represents the k-th communication characteristic sample CM _km of the sample vector x _m in the terminal device m. Therefore, the right side of formula (7) means that the weight w _k is calculated using M k-th communication characteristics CM _k1 , CM _k2 , ..., CM _km , ..., CM _kM shown in formula (5).

Thereafter, the extraction means 14 substitutes the weight w _k calculated by the formula (7) into the following formula to calculate the rank weight R _k for determining the rank.

The extraction means 14 calculates the rank weights R _k for all of the K communication characteristics by the above-mentioned method.

Fig. 16 is a diagram showing the calculation result of the rank weight _Rk . Referring to Fig. 16, the rank weight _Rk and the rank are associated with the maximum scan delay _DLmaxm , the jitter _JTm , the minimum scan delay _DLminm , and the packet loss rate _PLRm .

Then, the extraction means 14 determines the communication characteristic (here, the packet loss rate PLR _m ) with the lowest rank (i.e., with the smallest rank weight R _k ) as the communication characteristic to be deleted. Thereafter, the extraction means 14 deletes samples of the communication characteristic from the sample vectors x ₁ to x _M , and repeats the calculations of the equations (4), (6) to (8) and the determination of the communication characteristic to be deleted until all communication characteristics are deleted. Note that, if there are multiple communication characteristics with the smallest rank weight R _k , the extraction means 14 deletes any one of these communication characteristics. Also, if there are multiple communication characteristics with the smallest rank weight R _k , the extraction means 14 may determine the communication characteristic with the smallest correlation coefficient r _ij with the scan response delay RTT _m as the communication characteristic to be deleted.

Then, the extraction means 14 detects the last deleted communication characteristic as the communication characteristic CMbest _m _unknown, combines the detected communication characteristic CMbest _m _unknown with the scan response delay RTT _m _unknown, and outputs the combined result to the estimation means 16 .

In addition, when a network scan is performed by monitoring background traffic according to the schedule SCH_scan shown in Figure 6, the extraction means 14 selects the communication characteristic CMbest m _unknown by the above-mentioned method based on the M scan response delays RTT _m , M packet loss rates PLR _m , M delays DL m, M jitters JT _m , M background maximum delays BGDLmax _m , _and M background minimum delays BGDLmin _m , instead of the above-mentioned M scan response delays RTT _m , M packet loss rates PLR _m , M scan maximum delays DLmax _{m ,} and M scan minimum delays DLmin _m .

[Selection of the scan response delay RTT _m _known that is most similar to the scan response delay RTT _m _unknown]
The extraction means 14 detects the scan response delays RTT ₁ to RTT _M as scan response delays RTT ₁ _unknown to RTT _M _unknown from the correspondence table TBL1 (or correspondence table TBL2) stored in the host information database 13. In this case, each of the scan response delays RTT ₁ to RTT _M is composed of one scan response delay or an average value of multiple scan response delays within one IP address Add _m .

In addition, the extraction means 14 detects scan response delay {RTT _m' } _known _1 (m' = 1' to M'), scan response delay {RTT _m' } _known _2 (m' = 1' to M') and scan response delay {RTT _m' } _known _3 (m' = 1' to M') from the correspondence table TBL3 stored in the host information database 13.

Then, the extraction means 14 outputs the scan response delays RTT ₁ _unknown to RTT _M _unknown, the scan response delay {RTT _m' _known} _1 (m'=1' to M'), the scan response delay {RTT _m' _known} _2 (m'=1' to M') and the scan response delay {RTT _m' _known} _3 (m'=1' to M') to the calculation means 15.

Calculation means 15 receives scan response delays RTT ₁ _unknown to RTT _M _unknown, scan response delay {RTT _m' _known} _1 (m'=1' to M'), scan response delay {RTT _m' _known} _2 (m'=1' to M') and scan response delay {RTT _m' _known} _3 (m'=1' to M') from extraction means 14 .

Calculation means 15 acquires mode m (Mo), which is the most likely value of the scan response delay, based on scan response delay RTT _m . When there are two most likely values of the scan response delay, calculation means 15 calculates mode m (Mo), which is the most likely value of the scan response delay, by the following formula. When there are three or more most likely values of the scan response delay, calculation means 15 detects minimum and maximum values from the three or more most likely scan response delays, and calculates mode m (Mo), which is the most likely value of the scan response delay, by the following formula based on the detected minimum and maximum values.

In formula (9), L indicates the lower boundary of the probability density function region (=PDF (Probability Density Function) region) including two values of the most likely scan response delay when the result of the network scan is unknown or when the result of the network scan is known. Furthermore, r indicates the number of samples (=the number of scan response delays) of the most likely PDF region when the result of the network scan is unknown or when the result of the network scan is known. Furthermore, r ₁ indicates the number of samples (=the number of scan response delays) of the previous PDF region in the direction where the scan response delay RTT _m is smaller than the most likely PDF region when the result of the network scan is unknown or when the result of the network scan is known. Furthermore, r ₂ indicates the number of samples (=the number of scan response delays) of the next PDF region in the direction where the scan response delay RTT _m is larger than the most likely PDF region when the result of the network scan is unknown or when the result of the network scan is known. Furthermore, h indicates the interval between each PDF region (i.e., the width of each PDF region).

Figures 17 to 19 show specific distributions of scan response delays. Figure 20 shows the correspondence between PDF regions and the number of samples generated. Figure 21 shows a specific distribution of the probability density function of scan response delays.

When the number of samples for each specific value in the specific distribution of scan response delay shown in Figure 17 is counted and the number of samples for each specific value is displayed, the result is as shown in Figure 18. As a result, Figure 18 shows that scan response delay is most likely to occur at probability density values of 0.41 and 0.6.

As shown in Figure 19, the number of samples generated in the PDF region of 0.0 to 0.2, the PDF region of 0.21 to 0.4, the PDF region of 0.41 to 0.6, the PDF region of 0.61 to 0.8, and the PFD region of 0.81 to 1.0 are "5", "5", "14", "4", and "2", respectively.

The correspondence between the PDF regions and the number of samples generated in each region is as shown in Figure 20. The number of samples generated in the PDF region of 0.41 to 0.6 (=14) is r, the interval of 0.41 to 0.6 (=0.2) is h, 0.41 is L, the number of samples generated in the PDF region of 0.21 to 0.4 (=5) is _r1 , and the number of samples generated in the PDF region of 0.61 to 0.8 (=4) is _r2 . Therefore, the mode m (Mo) can be calculated by substituting r (=14), _r1 (=5), _r2 (=4), h (=0.2), and L (=0.41) into formula (9).

As shown in FIG. 21, r is the number of samples of the scan response delay in the PDF region REG _PDF surrounded by the two values (=0.41 and 0.6) that are most likely to cause the scan response delay and the curve showing the probability density function. Furthermore, L is the lower value (=0.41) of the two values (=0.41 and 0.6) that are most likely to cause the scan response delay. Furthermore, h is the interval between the two values (=0.41 and 0.6). r ₁ is the number of samples in a region of width h adjacent to the PDF region REG _PDF on the side where the scan response delay is smaller than the _PDF region REG _PDF , and r ₂ is the number of samples in a region of width h adjacent to the PDF region REG PDF on the side where the scan response delay is larger than the PDF region REG _PDF .

FIG. 22 is a diagram showing another specific distribution of scan response delay. Referring to FIG. 22, there are three most likely scan response delay values. In this case, the interval h is adjusted to the width of the PDF region including the three most likely scan response delay values. Then, the minimum value (=0.35) and the maximum value (=0.6) are detected from the three most likely scan response delay values. After that, the minimum value (=0.35) and the maximum value (=0.6) are used as the two most likely scan response delay values to determine r, r ₁ , r ₂ , L and h by the method described in FIG. 17 to FIG. 21, and the determined r, r ₁ , r ₂ , L and h are substituted into formula (9) to calculate the mode m (Mo).

When there are four or more most likely scan response delay values, the mode m (Mo) is calculated in the same manner as when there are three most likely scan response delay values.

Hereinafter, the mode m(Mo) calculated using equation (9) based on the scan response delays RTT ₁ _unknown to RTT _M _unknown will be denoted as “m(Mo)_unknown”, the mode m(Mo) calculated using equation (9) based on the scan response delays {RTT _m′ _known} _1 (m′=1′ to M′) will be denoted as “m(Mo)_known_1”, the mode m(Mo) calculated using equation (9) based on the scan response delays {RTT _m′ _known} _2 (m′=1′ to M′) will be denoted as “m(Mo)_known_2”, and the mode m(Mo) calculated using equation (9) based on the scan response delays {RTT _m′ _known}_3 (m'=1' to M'), the mode m(Mo) calculated by equation (9) is represented as "m(Mo)_known_3".

The calculation means 15 calculates the modes m(Mo)_unknown, m(Mo)_known_1, m(Mo)_known_2, and m(Mo)_known_3 by the above-mentioned method, and then calculates the mean absolute deviation W _unknown of the scan response delays RTT ₁ _unknown to RTT _M _unknown by the following equation.

C(x _{i — unknown} ) in equation (10A) is expressed by equation (10B), and scan response delays RTT ₁ _unknown to RTT _M _unknown are substituted for x _{i _unknown} .

After calculating the mean absolute deviation W _unknown , the calculation means 15 calculates the mean absolute deviation W _{known_z} of the scan response delay {RTT _m' _known}_z (m'=1' to M') by the following equation.

In formula (11), z is an argument indicating the known estimated results of the network scan, and is an integer satisfying 1≦z≦Z. Furthermore, Z is the total number of known estimated results of the network scan. In this embodiment of the invention, the known estimated results of the network scan are "normal", "poor communication quality", and "network congestion", so in formula (11), z = 1, 2, 3.

The scan response delays corresponding to the known estimated network scan results of "normal", "poor communication quality" and "network congestion" may consist of one type of data or multiple types of data.

The calculation means 15 substitutes the mode m(Mo)_known_1 into m(Mo)_known_z in equation (11B), and substitutes the scan response delays {RTT _1′ _known}_1 to {RTT _M′ _known}_1 into x _{i _known_z} in equation (11B), to calculate the mean absolute deviation W _{known_1} according to equation (11A).

In addition, the calculation means 15 substitutes the mode m(Mo)_known_2 into m(Mo)_known_z in equation (11B) and substitutes the scan response delays {RTT _1′ _known}_2 to {RTT _M′ _known}_2 into x _{i _known_z} in equation (11B) to calculate the mean absolute deviation W _{known_2} by equation (11A).

Furthermore, the calculation means 15 substitutes the mode m(Mo)_known_3 into m(Mo)_known_z in equation (11B), and substitutes {RTT _{1 '} _known}_3 to {RTT _{M'_known} }_3 into x _{i _known_z} in equation (11B), to calculate the mean absolute deviation W _{known_3} according to equation (11A).

That is, the calculation means 15 calculates the mean absolute deviations W known_1, W known_2, and _W known_3 based on the modes m(Mo)_known_1, m(Mo)_known_2, and m(Mo)_known_3 and the scan response delays {RTT _{1 '} _known}_1 to {RTT _M' _known}_1, {RTT _1' _known}_2 to { _{RTT M'} _known}_2, and {RTT 1' _{_known} }_3 to {RTT M' _known} _{_3} using equation (11).

Thereafter, the calculation means 14 substitutes the mean absolute deviations W _unknown , W _{known — 1} , W _{known — 2} , and W _{known — 3} into the following equation to calculate the differences S _S1 , S _S2 , and S _S3 in the mean absolute deviations.

In this case, the calculation means 15 substitutes the mean absolute deviation W _{known —} 1 for W _{known — z} in equation (12) and substitutes the mean absolute deviation W _unknown for W _unknown in equation (12) to calculate the difference S _S1 in the mean absolute deviations.

Furthermore, the calculation means 15 substitutes the mean absolute deviation W _{known — 2} into W _{known — z} in equation (12) and substitutes the mean absolute deviation W _unknown into W _unknown in equation (12) to calculate the difference S _S2 in the mean absolute deviations.

Furthermore, the calculation means 15 substitutes the mean absolute deviation W _{known — 3} into W _{known — z} in equation (12) and substitutes the mean absolute deviation W _unknown into W _unknown in equation (12) to calculate the difference S _S3 in the mean absolute deviations.

After calculating the differences S _S1 , S _S2 , and S _S3 in the mean absolute deviations, the calculation means 15 substitutes the calculated differences S _S1 , S _S2 , and S _S3 in the mean absolute deviations into the following equation to calculate the similarity P _Sz .

In this case, the calculation means 15 calculates the similarity P _S1 by substituting the difference S _S1 in the mean absolute deviations into S _Sz in formula (13), calculates the similarity P S2 by substituting the difference S _S2 in the mean absolute deviations into S _Sz in formula (13), and calculates the similarity P _S3 by substituting the difference S _S3 in the mean absolute deviations into S _Sz in formula (13 ₎ .

After calculating the similarities P _S1 , P _S2 , and P _S3 , the calculation means 15 outputs the calculated similarities P _S1 , P _S2 , and P _S3 to the extraction means 14 .

When the extraction means 14 receives the similarities P _S1 , P _S2 , and P _S3 from the calculation means 15, it detects the highest similarity P _{S_highest} among the similarities P _S1 , P _S2 , and P _S3 . Then, the extraction means 14 refers to the correspondence table TBL3 stored in the host information database 13 and detects the _scan response delay {RTT _m' _known}_similarity (m'=1' to M') (= any of scan response delay {RTT _m' _known}_1 (m'=1' to M'), scan response delay {RTT _m' _known}_2 (m'=1' to M'), and scan response delay {RTT _m' _known}_3 (m'=1' to M')) when the highest similarity P S_highest is obtained. In addition, the extraction means 14 detects a communication characteristic {CMbest _m' _known} _similarity (m' = 1' to M') (= any one of communication characteristic {CMbest _m' _known} _1 (m' = 1' to M'), communication characteristic {CMbest _m' _known} _2 (m' = 1' to M'), and communication characteristic {CMbest _m' _known} _3 (m' = 1' to M')) used to estimate a known estimation result of a network scan together with a scan response delay {RTT _m' _known} _similarity (m' = 1' to M').

Then, the extraction means 14 generates standard sample data D_similarity consisting of the scan response delay {RTT _m' _known}_similarity (m' = 1' to M') and the communication characteristic {CMbest _m' _known}_similarity (m' = 1' to M'), and outputs the generated standard sample data D_similarity to the estimation means 16.

In addition, the extraction means 14 detects scan response delays RTT ₁ _unknown to RTT _M _unknown (= scan response delays RTT _m (m = 1 to M)) by referring to correspondence table TBL1 (or correspondence table TBL2) stored in the host information database 13, generates sample data D_unknown consisting of the detected scan response delays RTT ₁ _unknown to RTT _M _unknown (= scan response delays RTT _m (m = 1 to M)) and communication characteristics CMbest_unknown_m (m = 1 to M), and outputs the generated sample data D_unknown to the estimation means 16.

The estimation means 16 receives the standard sample data D_similarity and the sample data D_unknown from the extraction means 14.

The estimation means 16 then updates the estimation algorithm ALG to the optimal algorithm using the standard sample data D_similarity in the manner described below.

Figure 23 is a diagram showing data when training the estimation algorithm ALG. With reference to Figure 23, when training the estimation algorithm ALG using the standard sample data D_similarity, the estimation means 16 divides the standard sample data D_similarity into training data and test data. In this case, the estimation means 16 divides the standard sample data D_similarity into training data and test data so that the proportion of training data is 80% and the proportion of test data is 20%, for example. Note that when dividing the standard sample data D_similarity into training data and test data, the proportion of training data may be other than 80%, and the proportion of test data may be other than 20%.

Then, the estimation means 16 divides the training data into a training set and a validation set. In this case, the estimation means 16 divides the training data into a training set and a validation set so that the proportion of the training set is 50% and the proportion of the validation set is 50%, for example. Note that the proportions of the training set and the validation set may be other than 50%.

The estimation means 16 divides the training data into a training set and a validation set, and then divides the training set into U subsets d ₁ to d _U. Each of the subsets d ₁ to d _U is made up of data of size n. Here, n is, for example, 8. The estimation means 16 also divides the validation set into U' subsets V _1' to V _U' . Then, if the data size of the training set is D _S , then U=INT(D _S /n).

Fig. 24 is a conceptual diagram of sample data when training the estimation algorithm ALG. Referring to Fig. 24, when training the estimation algorithm ALG, the estimation means 16 creates initial sample data for modeling. More specifically, estimation means 16 reads out scan response delays {RTT _1' _known}_1 to {RTT _M' _known}_1 and communication characteristics {CMbest _1' _known}_1 to {CMbest _M' _known}_1 from correspondence table TBL3 shown in FIG. 3, and generates standard sample data D_standard_1 from the read scan response delays {RTT _1' _known}_1 to {RTT _M' _known}_1 and communication characteristics {CMbest _1' _known}_1 to {CMbest _M' _known}_1.

In addition, estimation means 16 reads out scan response delays {RTT _1' _known}_2 to {RTT _M' _known}_2 and communication characteristics {CMbest ₁ ' _known}_2 to {CMbest _M' _known}_2 from correspondence table TBL3, and generates standard sample data D_standard_2 from the read scan response delays {RTT _1' _known}_2 to {RTT _M' _known}_2 and communication characteristics {CMbest _1' _known}_2 to {CMbest _M' _known}_2.

Furthermore, estimation means 16 reads out scan response delays {RTT _1' _known}_3 to {RTT _M' _known}_3 and communication characteristics {CMbest _1' _known}_3 to {CMbest _M' _known}_3 from correspondence table TBL3, and generates standard sample data D_standard_3 from the read scan response delays {RTT _1' _known}_3 to {RTT _M' _known}_3 and communication characteristics {CMbest _1' _known}_3 to {CMbest _M' _known}_3.

Then, the estimation means 16 generates the initial sample data D_sample_initial as follows (see FIG. 24(a)). In this case, the estimation means 16 selects one subset d 1 _{_initial} from the multiple subsets d _{1 _1} to d _{U _1} of the standard sample data D_standard_1, selects one subset d 2 _initial from the multiple subsets d _{1 _2} to d _{U _2} of the standard sample data _{D_standard_2} , and selects one subset d 3 _{_initial} from the multiple subsets d _{1 _3} to d _{U _3} of the standard sample data D_standard_3. Then, "1", "2", and "3" in the standard sample data D_standard_1, D_standard_2, and D_standard_3 respectively represent the identification code ID_code=1 indicating that the result of the network scan is "normal", the identification code ID_code=2 indicating that the result of the network scan is "insufficient communication quality", and the identification code ID_code=3 indicating that the result of the network scan is "network congestion", and therefore the estimation means 16 selects the subsets _{d1_initial} , _{d2_initial} , and _{d3_initial} , and combines the subsets _{d1_initial} , d2_initial, and _{d3_initial} via the identification codes ID_code=1, ₂ , and 3 to generate initial sample data D_sample_initial.

In addition, "1" of the multiple subsets d _{1_1} to d _{U_1} corresponds to "1" of the standard sample data D_standard_1, "2" of the multiple subsets d _{1_2} to d _{U_2} corresponds to "2" of the standard sample data D_standard_2, and "3" of the multiple subsets d _{1_3} to d _{U_3} corresponds to "3" of the standard sample data D_standard_3. In addition, one subset d _{1_initial} , d _{2_initial} , d 3_initial may be selected randomly from the multiple subsets d _{1_1} to d _{U_1} , the multiple subsets d _{1_2 to} d _{U_2} , and the multiple subsets d _{1_3} to d _{U_3} , respectively, or may be selected according to any selection method.

23, the subsets d ₁ to d _U are generated from the standard sample data D_similarity. The standard sample data D_similarity is the sample data that has the highest similarity to the sample data D_unknown among the standard sample data D_standard_1 to D_standard_3.

As a result, the subsets d ₁ to d _U are generated from the standard sample data D_standard_1 by the method described in Fig. 23 when the standard sample data D_standard_1 has the highest similarity to the sample data D_unknown. Also, the subsets d ₁ to d _U are generated from the standard sample data D_standard_2 by the method described in Fig. 23 when the standard sample data D_standard_2 has the highest similarity to the sample data D_unknown. Furthermore, the subsets d ₁ to d _U are generated from the standard sample data D_standard_3 by the method described in Fig. 23 when the standard sample data D_standard_3 has the highest similarity to the sample data D_unknown.

When the subsets d ₁ to d _U are generated from the standard sample data D_standard_1, the standard sample data D_standard_1 in the initial sample data D_sample_initial for modeling is replaced by the subset d _u to generate tentative sample data D_sample_tentative (D1) for modeling (see FIG. 24B). Note that "D1" in D_sample_tentative (D1) indicates that the standard sample data D_standard_1 in the initial sample data D_sample_initial has been replaced by the subset d _u .

Furthermore, when the subsets d ₁ to d _U are generated from the standard sample data D_standard_2, the standard sample data D_standard_2 in the initial sample data D_sample_initial for modeling is replaced by the subset d _u to generate tentative sample data D_sample_tentative (D2) for modeling (see FIG. 24(c)). Note that "D2" in D_sample_tentative (D2) indicates that the standard sample data D_standard_2 in the initial sample data D_sample_initial has been replaced by the subset d _u .

Furthermore, when the subsets d ₁ to d _U are generated from the standard sample data D_standard_3, the standard sample data D_standard_3 in the initial sample data D_sample_initial for modeling is replaced by the subset d _u to generate tentative sample data D_sample_tentative (D3) for modeling (see FIG. 24(d)). Note that "D3" in D_sample_tentative (D3) indicates that the standard sample data D_standard_3 in the initial sample data D_sample_initial has been replaced by the subset d _u .

Fig. 25 is a conceptual diagram showing an update of tentative sample data for modeling. In Fig. 25, a method of updating tentative sample data for modeling D_sample_tentative (D1) is described using an example in which subsets d ₁ to d _U are generated from standard sample data D_standard_1.

25, the estimation means 16 first creates initial sample data D_sample_initial for modeling by the above-mentioned method. After that, the estimation means 16 replaces the standard sample data D_standard_1 of the sample data D_sample_initial by the subset d ₁ to update the initial sample data D_sample_initial to tentative sample data D_sample_tentative_1 for modeling. Then, by updating the initial sample data D_sample_initial to the tentative sample data D_sample_tentative_1, the initial sample data D_sample_initial becomes the "old training sample" and the tentative sample data D_sample_tentative_1 becomes the "new training sample".

Then, if the estimation means 16 determines that the evaluation index when estimating the results of the network scan using the tentative sample data D_sample_tentative_1 (= "new training sample") is better than the evaluation index when estimating the results of the network scan using the initial sample data D_sample_initial (= "old training sample"), the tentative sample data D_sample_tentative_1 becomes the "old training sample."

Next, the estimation means 16 updates the tentative sample data D_sample_tentative_1 to tentative sample data D_sample_tentative_2 by replacing the subset d ₁ of the tentative sample data D_sample_tentative_1 with the subset d _2. Then, if the evaluation index when the result of the network scan is estimated using the tentative sample data D_sample_tentative_2 (= "new training sample") is better than the evaluation index when the result of the network scan is estimated using the "old training sample", the tentative sample data D_sample_tentative_2 becomes the "old training sample". In addition, when comparing the evaluation index between the tentative sample data D_sample_tentative_2 (= "new training sample") and the "old training sample", the "old training sample" is made up of the tentative sample data D_sample_tentative_1 if, in the previous comparison of evaluation indices, the evaluation index when the result of the network scan was estimated using the tentative sample data D_sample_tentative_1 was better than the evaluation index when the result of the network scan was estimated using the initial sample data D_sample_initial, and is made up of the initial sample data D_sample_initial if the evaluation index when the result of the network scan was estimated using the initial sample data D_sample_initial was better than the evaluation index when the result of the network scan was estimated using the tentative sample data D_sample_tentative_1. In other words, the sample data with the better evaluation index will become the "old training sample" to be compared in the next comparison of evaluation indexes.

Furthermore, the estimation means 16 continues by replacing the subset d ₂ of the tentative sample data D_sample_tentative_2 with the subset d ₃ , thereby updating the tentative sample data D_sample_tentative_2 to the tentative sample data D_sample_tentative_3. Then, if the evaluation index when the result of the network scan is estimated using the tentative sample data D_sample_tentative_3 is better than the evaluation index when the result of the network scan is estimated using the "old training sample", the tentative sample data D_sample_tentative_3 becomes the "old training sample".

The estimation means 16 then repeats the process of updating the provisional sample data D_sample_tentative_u to the provisional sample data D_sample_tentative_u+1 in a similar manner, and finally updates it to the provisional sample data D_sample_tentative_U.

In this way, the estimation means 16 successively replaces the standard sample data D_standard — 1 of the initial sample data D_sample_initial with U subsets d ₁ to d _U , thereby successively updating the initial sample data D_sample_initial to tentative sample data D_sample_tentative — 1 to D_sample_tentative_U.

Then, when the standard sample data D_standard_1 has the highest similarity to the sample data D_unknown, the initial sample data D_sample_initial and the tentative sample data D_sample_tentative_1 to D_sample_tentative_U are used to train the estimation algorithm ALG.

When the standard sample data D_standard_2 has the highest similarity to the sample data D_unknown, the tentative sample data D_sample_tentative (D2) shown in (c) of FIG. 24 constitutes the tentative sample data D_sample_tentative_1, and the estimation means 16 sequentially updates the initial sample data D_sample_initial to tentative sample data D_sample_tentative_1 to D_sample_tentative_U by the method described in FIG. 25.

Furthermore, when the standard sample data D_standard_3 has the highest similarity to the sample data D_unknown, the tentative sample data D_sample_tentative (D3) shown in (d) of Figure 24 constitutes the tentative sample data D_sample_tentative_1, and the estimation means 16 sequentially updates the initial sample data D_sample_initial to the tentative sample data D_sample_tentative_1 to D_sample_tentative_U by the method described in Figure 25.

As described above, the initial sample data D_sample_initial is composed of the standard sample data D_standard_1 to D_standard_3 combined, and the tentative sample data D_sample_tentative_1 to D_sample_tentative_U are composed of any of the standard sample data D_standard_1 to D_standard_3 in the initial sample data D_sample_initial replaced by a subset d ₁ to d _U , depending on which of the standard sample data D_standard_1 to D_standard_3 the standard sample data D_similarity is composed of. The subsets d ₁ to d _U are each composed of any one of the standard sample data D_standard_ 1 to D_standard_ 3 that make up the standard sample data D_similarity.

Therefore, the initial sample data D_sample_initial and the tentative sample data D_sample_tentative_1 to D_sample_tentative_U are basically composed of a combination of parts of the standard sample data D_standard_1 to D_standard_3.

The initial sample data D_sample_initial and the tentative sample data D_sample_tentative_1 to D_sample_tentative_U are composed of the standard sample data D_standard_1 to D_standard_3 combined for the following reasons.

As described later, the estimation algorithm ALG classifies a group _CLS1 in which the result of the network scan is “normal”, a group _CLS2 in which the result of the network scan is “inadequate communication quality”, and a group CLS3 in which the result of the network scan is “network congestion”. ₃ , the initial sample data D_sample_initial and the tentative sample data D_sample_tentative_1 to D_sample_tentative_U, which are data used when training the estimation algorithm ALG, are composed of standard sample data D_standard_1 acquired in a communications environment where the network scan result is "normal", standard sample data D_standard_2 acquired in a communications environment where the network scan result is "inadequate communication quality", and standard sample data D_standard_3 acquired in a communications environment where the network scan result is "network congestion", so that the network scan result can be accurately estimated as any of "normal", "inadequate communication quality", and "network congestion".

Figure 26 is a conceptual diagram of training and evaluation of the estimation algorithm ALG. Referring to Figure 26, the estimation means 16 trains the estimation algorithm ALG using the initial sample data D_sample_initial. In this case, the estimation means 16 trains the estimation algorithm ALG by repeatedly estimating the results of the network scan by the estimation algorithm ALG using the initial sample data D_sample_initial.

Then, the estimation means 16 estimates the results of the network scan using the validation set. After that, the estimation means 16 evaluates the estimated results of the network scan using the evaluation index EVD.

Subsequently, the estimation means 16 performs the following steps for all of the tentative sample data D_sample_tentative_1 to D_sample_tentative_U: (I) training the estimation algorithm ALG using the tentative sample data D_sample_tentative_u; (II) estimating the results of the network scan using a validation set (= the entire subsets V1 _' to VU _' ); and (III) evaluating the estimated results of the network scan using an evaluation index nEVD_u. In this case, the estimation means 16 determines whether the evaluation index nEVD_u is greater than the evaluation index EVD each time it executes (I) to (III), and when the evaluation index nEVD_u is greater than the evaluation index EVD, it sets the evaluation index nEVD_u to the evaluation index EVD and updates the evaluation index EVD with the evaluation index nEVD_u, and when the evaluation index nEVD_u is not greater than the evaluation index EVD, it does not update the evaluation index EVD (i.e., it maintains the evaluation index EVD). Then, the estimation means 16 maintains the variables of the estimation algorithm ALG when the evaluation index EVD is the best when the result of the network scan is estimated using the initial sample data D_sample_initial, and when the evaluation index nEVD_u is the best when the result of the network scan is estimated using the hypothetical sample data (any of the hypothetical sample data D_sample_tentative_1 to D_sample_tentative_U), replaces the variables of the estimation algorithm ALG with the hypothetical sample data (any of the hypothetical sample data D_sample_tentative_1 to D_sample_tentative_U) when the largest evaluation index nEVD_u is obtained.

Then, the estimation means 16 estimates the results of the network scan using the test data, and evaluates whether to approve the training model (estimation algorithm ALG after training) and the training dataset based on the evaluation index tEVD. More specifically, when the evaluation index tEVD is equal to or greater than the evaluation index EVD (finally updated evaluation index EVD), the estimation means 16 approves the training model (estimation algorithm ALG after training) and the training dataset, and when the evaluation index tEVD is not equal to or greater than the evaluation index EVD (finally updated evaluation index EVD), the estimation means 16 does not approve the training model (estimation algorithm ALG after training) and the training dataset. Then, when the estimation means 16 does not approve the training model (estimation algorithm ALG after training) and the training dataset, (I) to (IV) shown in FIG. 26 are repeatedly executed using the estimation algorithm ALG with the variables finally replaced.

The approved training data set becomes the initial sample data D_sample_initial when the next training and evaluation are performed. The evaluation index EVD is composed of the F1 score _F1 or the Matthews correlation coefficient MCC described later, the evaluation index nEVD_u is composed of the F1 score _nF1 or the Matthews correlation coefficient nMCC described later, and the evaluation index tEVD is composed of the F1 score _tF1 or the Matthews correlation coefficient tMCC described later.

In this way, the estimation means 16 trains the estimation algorithm ALG using the initial sample data D_sample_initial and the tentative sample data D_sample_tentative_1 to D_sample_tentative_U, and evaluates the estimation results when the network scan results are estimated by the trained estimation algorithm ALG using a validation set (= consisting of all of the subsets V1 _' to VU _' ) using the evaluation indexes EVD and nEVD, and finally, uses the test data to evaluate whether the trained estimation algorithm ALG and the training data set are to be approved using the evaluation index tEVD.

Figure 27 is a diagram showing a specific example of the initial sample data D_sample_initial for modeling shown in Figure 25. Figure 28 is a diagram showing a specific example of the training set shown in Figure 23. Figure 29 is a diagram showing a specific example of the validation set shown in Figure 23. Figure 30 is a diagram showing a specific example of the test data shown in Figure 23.

27 to 30, a specific example of the initial sample data _{D_sample_initial} , a specific example of the training set, a specific example of the validation set, and a specific example of the test data are shown, using an example in which the number of terminal devices to be subjected to the network scan is 8 (i.e., M′=8) and the number of samples of the standard sample data D_similarity is 20 for one IP address Add m′.

27, the standard sample data D_standard_1 is composed of scan response delays RTT _m' and communication characteristics CMbest _m' when a network scan is performed in a communication environment where the result of the network scan is "normal." Therefore, in the standard sample data D_standard_1, the eight specific values of the scan response delays RTT _m' and the eight specific values of the communication characteristics CMbest _m' are a subset of the standard sample data D_standard_1 when M'=8 in the scan response delays {RTT _1' _known}_1 to {RTT _M' _known}_1 stored in the correspondence table TBL3 shown in FIG.

Moreover, the standard sample data D_standard_2 is composed of scan response delays RTT _m' and communication characteristics CMbest _m' when a network scan is performed in a communication environment where the result of the network scan is "insufficient communication quality." Therefore, in the standard sample data D_standard_2, the eight specific values of the scan response delays RTT _m' and the eight specific values of the communication characteristics CMbest _m' are a subset of the standard sample data D_standard_2 when M'=8 in the scan response delays {RTT _1' _known}_2 to {RTT _M' _known}_2 stored in the correspondence table TBL3.

Furthermore, the standard sample data D_standard_3 is composed of scan response delays RTT _m' and communication characteristics CMbest _m' when a network scan is performed in a communication environment where the result of the network scan is "network congestion." Therefore, in the standard sample data D_standard_3, the eight specific values of the scan response delays RTT _m' and the eight specific values of the communication characteristics CMbest _m' are a subset of the standard sample data D_standard_3 when M'=8 in the scan response delays {RTT _1' _known}_3 to {RTT _M' _known}_3 stored in the correspondence table TBL3.

As a result, the initial sample data D_sample_initial for modeling is composed of the standard sample data D_standard_1 to D_standard_3 combined with the identification code D_code=1 assigned to the network scan result "normal," the identification code D_code=2 assigned to the network scan result "poor communication quality," and the identification code D_code=3 assigned to the network scan result "network congestion."

28, the training set consists of eight subsets d ₁ to d _8. Since M′=8, each of the subsets d ₁ to d ₈ consists of eight scan response delays RTT _1′ to RTT _8′ and eight communication characteristics CMbest _1′ to CMbest _8′ . Therefore, in each of the subsets d ₁ to d ₈ , the vertical direction indicates eight scan response delays RTT _1′ to RTT _8′ corresponding to eight terminal devices and eight communication characteristics CMbest _1′ to CMbest _8′ .

The number of subsets d ₁ to d ₈ is eight because the number of samples in the standard sample data D_similarity is 20, which is 20×0.8×0.5=8.

5. Furthermore, when the standard sample data D_similarity is made up of the standard sample data D_standard_1, each of the subsets d ₁ to d ₈ is made up of the scan response delays {RTT _{1 ′} _known}_1 to {RTT _M′ _known}_1 and the communication characteristics {CMbest _{1 ′} _known}_1 to {CMbest _M′ _known}_1 stored in the correspondence table TBL3 shown in FIG. 5. When the standard sample data D_similarity is made up of the standard sample data D_standard_2, each of the subsets d 1 to d 8 is made up of the scan response delays {RTT _{1 ′} _known}_2 to {RTT _M′ _known}_2 and the communication characteristics {CMbest _{1 ′} _known}_2 to {CMbest _M′ _known}_2 and the standard sample data D_similarity is standard sample data D_standard_3, the correspondence table TBL3 shown in FIG. 5 contains scan response delays {RTT _1' _known}_3 to {RTT _M' _known}_3 and communication characteristics {CMbest _1' _known}_3 to {CMbest _M' _known}_3.

Furthermore, when the subsets d ₁ to d ₈ each consist of scan response delays {RTT _1′ _known}_1 to {RTT _8′ _known}_1 and communication characteristics {CMbest _1′ _known}_1 to {CMbest _8′ _known}_1, the standard sample data D_standard_1 of the initial sample data D_sample_initial shown in FIG. 27 is sequentially replaced with the subsets d ₁ to d ₈ , thereby sequentially generating tentative sample data D_sample_tentative_1 to D_sample_tentative_U (U=8) shown in FIG. 25.

Furthermore, when the subsets d ₁ to d ₈ each consist of scan response delays {RTT _{1 '} _known} _2 to {RTT _{8 '} _known} _2 and communication characteristics {CMbest ₁ '_known} _22 to {CMbest _{8 '} _known} _2, the standard sample data D_standard_2 of the initial sample data D_sample_initial shown in FIG. 27 is replaced sequentially with the subsets d ₁ to d ₈ , thereby sequentially generating tentative sample data D_sample_tentative_1 to D_sample_tentative_U (U = 8) shown in FIG. 25.

Furthermore, when the subsets d ₁ to d ₈ each consist of scan response delays {RTT _1′ _known} _3 to {RTT _8′ _known} _3 and communication characteristics {CMbest _1′ _known} _3 to {CMbest _8′ _known} _3, the standard sample data D_standard_3 of the initial sample data D_sample_initial shown in FIG. 27 is sequentially replaced with the subsets d ₁ to d ₈ , thereby sequentially generating tentative sample data D_sample_tentative_1 to D_sample_tentative_U (U=8) shown in FIG. 25.

29, the validation set consists of eight subsets V _{1 '} to V _{8 '} . Since M'=8, each of the subsets V _{1 '} to V _{8 '} consists of eight scan response delays RTT _{1 '} to RTT _{8 '} and eight communication characteristics CMbest _{1 '} to CMbest _{8 '} . Therefore, in each of the subsets V _{1 '} to V _{8 '} , the vertical direction indicates eight scan response delays RTT _{1 '} to RTT _{8 '} corresponding to eight terminal devices and eight communication characteristics CMbest _{1 '} to CMbest _{8 '} .

The reason why the number of subsets V _{1 '} to V _{8 '} is eight is the same as the reason why the number of subsets d ₁ to d ₈ is eight.

5. Furthermore, when the standard sample data D_similarity is made up of the standard sample data D_standard_1, each of the subsets V _{1 ′} to V _{8 ′} is made up of a part of the scan response delays {RTT _{1 ′} _known} _1 to {RTT _M′ _known} _1 and a part of the communication characteristics {CMbest _{1 ′} _known} _1 to {CMbest _{M′ _known} _1 stored in the correspondence table TBL3 shown in FIG. 5. When the standard sample data D_similarity is made up of the standard sample data D_standard_2, each of the subsets V 1 ′} to V 8 ′ is made up of a part of the scan response delays {RTT _{1 ′} _known} _2 to {RTT _M′ _known} _2 and a part of the communication characteristics { _CMbest 5. When the standard sample data D_similarity consists of a portion of the scan response delays {RTT _1' _known}_3 to {RTT _M' _known}_3 and a portion of the communication characteristics {CMbest 1' _known}_3 to {CMbest _{M' _known}_3, and when the standard sample data D_similarity consists of the standard sample data D_standard_3, the standard sample data D_similarity consists of a portion of the scan response delays {RTT 1' _known}_3 to {RTT M} ' _known}_3 and a portion of the communication characteristics {CMbest _1' _known}_3 to {CMbest _M' _known}_3.

30, the test data consists of four subsets TD ₁ to TD _4. Since M'=8, each of the subsets TD ₁ to TD ₄ consists of eight scan response delays RTT _1' to RTT _8' and eight communication characteristics CMbest _1' to CMbest _8' . Therefore, in each of the subsets TD ₁ to TD ₄ , the vertical direction indicates eight scan response delays RTT _1' to RTT _8' corresponding to eight terminal devices and eight communication characteristics CMbest _1' to CMbest _8' .

The reason why the number of subsets TD ₁ to TD ₄ is four is that the number of samples of the standard sample data D_similarity is 20 for one IP address Add _m' , and the test data consists of 20% (20×0.2=4) of the number of samples of the standard sample data D_similarity (=20).

5. Furthermore, when the standard sample data D_similarity is made up of the standard sample data D_standard_1, each of the subsets TD ₁ to TD ₄ is made up of a part of the scan response delays {RTT _{1 '} _known}_1 to {RTT _{M'_known} }_1 and a part of the communication characteristics {CMbest _{1 '} _known}_1 to {CMbest _{M'_known} }_1 stored in the correspondence table TBL3 shown in FIG. 5. When the standard sample data D_similarity is made up of the standard sample data D_standard_2, each of the subsets TD 1 to TD 4 is made up of a part of the scan response delays {RTT _{1 '} _known}_2 to {RTT _{M'_known} }_2 and a part of the communication characteristics {CMbest _{1 '} _known}_3 stored in the correspondence table TBL3 shown in FIG. 5. When the standard sample data D_similarity consists of a portion of the scan response delays {RTT _1' _known}_3 to {RTT _M' _known}_3 and a portion of the communication characteristics {CMbest 1' _known}_3 to {CMbest _{M' _known}_3, and when the standard sample data D_similarity consists of the standard sample data D_standard_3, the standard sample data D_similarity consists of a portion of the scan response delays {RTT 1' _known}_3 to {RTT M} ' _known}_3 and a portion of the communication characteristics {CMbest _1' _known}_3 to {CMbest _M' _known}_3.

The estimation means 16 trains an estimation algorithm ALG using the initial sample data D_sample_initial and the tentative sample data D_sample_tentative_1 to D_sample_tentative_U. When training the estimation algorithm ALG using the tentative sample data D_sample_tentative_1 to D_sample_tentative_U, the estimation means 16 sequentially generates tentative sample data D_sample_tentative_1 to D_sample_tentative_U (U=8) including subsets d ₁ to d ₈ , respectively, by the method described in Figures 25 and 28, and trains the estimation algorithm ALG by repeatedly estimating the results of the network scan by the estimation algorithm ALG using the generated tentative sample data D_sample_tentative_1 to D_sample_tentative_U (U=8).

The estimation means 16 trains the estimation algorithm ALG using the initial sample data D_sample_initial, then estimates the results of the network scan by the estimation algorithm ALG using the validation set, and generates a confusion matrix CFM based on the estimated results of the network scan.

The estimation means 16 also trains the estimation algorithm ALG using the tentative sample data D_sample_tentative_u, and then estimates the results of the network scan with the estimation algorithm ALG using the validation set, and generates a confusion matrix nCFM based on the estimated results of the network scan. The estimation means 16 performs this operation for all of the tentative sample data D_sample_tentative_1 to D_sample_tentative_U.

Figure 31 is a conceptual diagram of a confusion matrix. When estimating the results of a network scan, there are three known estimated results: "normal", "inadequate communication quality", and "network congestion". Therefore, in Figure 31, a confusion matrix is shown for the three known estimated results: "normal", "inadequate communication quality", and "network congestion".

With reference to Fig. 31, TP indicates the percentage of cases where an assumed event occurred and the occurrence of the assumed event (occurrence of the assumed event) was correctly estimated, TN indicates the percentage of cases where an assumed event did not actually occur and the occurrence of the assumed event (non-occurrence of the assumed event) was correctly estimated, FN indicates the percentage of cases where an assumed event occurred and the occurrence of the assumed event (occurrence of the assumed event) was incorrectly estimated as another event, and FP indicates the percentage of cases where an assumed event did not actually occur and the occurrence of the assumed event was incorrectly estimated as occurring. Also, in Fig. 31 (a), (b), and (c), "Normal", "Insufficient communication quality", and "Network congestion" in the column direction (vertical direction) represent events that actually occurred, and "Normal", "Insufficient communication quality", and "Network congestion" in the row direction (horizontal direction) represent estimated events. Furthermore, (a) in FIG. 31 shows the confusion matrix when the result of the network scan is "normal," (b) in FIG. 31 shows the confusion matrix when the result of the network scan is "inadequate communication quality," and (c) in FIG. 31 shows the confusion matrix when the result of the network scan is "network congestion."

In FIG. 31(a), the TP at the intersection of the "Normal" row and the "Normal" column indicates the percentage of cases where an assumed "normal" event (the occurrence of an assumed "normal" event) is correctly estimated as a "normal" event, the FN at the intersection of the "Normal" row and the "Poor communication quality" column indicates the percentage of cases where an assumed "normal" event is incorrectly estimated as a different "poor communication quality" event, and the FN at the intersection of the "Normal" row and the "Network congestion" column indicates the percentage of cases where an assumed "normal" event is incorrectly estimated as a different "network congestion" event.

In addition, the FP at the intersection of the row of "inadequate communication quality" and the column of "normal" indicates the percentage of cases where an event of "inadequate communication quality" that is not an assumed "normal" event occurred (i.e., the assumed "normal" event did not actually occur), and the TN at the intersection of the row of "inadequate communication quality" and the column of "inadequate communication quality" indicates the percentage of cases where an event of "inadequate communication quality" that is not an assumed "normal" event occurred, and the TN at the intersection of the row of "inadequate communication quality" and the column of "network congestion ...network congestion" that is not an assumed "normal" event occurred, and the TN at the intersection of the row of "inadequate communication quality" and the column of "network congestion" indicates the percentage of cases where an event of "inadequate communication quality" that is not an assumed "normal" event occurred, and the

Furthermore, the FP at the intersection of the "Network Congestion" row and the "Normal" column indicates the percentage of cases where a "Network Congestion" event that is not an assumed "normal" event occurred (i.e., the assumed "normal" event did not actually occur) was incorrectly estimated to have occurred as an assumed "normal" event, the TN at the intersection of the "Network Congestion" row and the "Inadequate Communication Quality" column indicates the percentage of cases where a "Network Congestion" event that is not an assumed "normal" event occurred was incorrectly estimated to have occurred as an "Inadequate Communication Quality" event (i.e., the assumed "normal" event did not actually occur) was correctly estimated, and the TN at the intersection of the "Network Congestion" row and the "Network Congestion" column indicates the percentage of cases where a "Network Congestion" event that is not an assumed "normal" event occurred was incorrectly estimated to have occurred (i.e., the assumed "normal" event did not actually occur) was correctly estimated.

In FIG. 31(b), the TN at the intersection of the “Normal” row and the “Normal” column indicates the percentage of cases where a “normal” event occurred that was not the assumed “inadequate communication quality” but a “normal” event occurred (i.e., when the assumed “inadequate communication quality” event did not actually occur, the percentage of cases where the assumption (the assumed “inadequate communication quality” event did not occur) was correctly assumed), and the FP at the intersection of the “Normal” row and the “inadequate communication quality” column indicates the percentage of cases where a “normal” event occurred that was not the assumed “inadequate communication quality” but a “normal” event occurred (i.e., when the assumed “inadequate communication quality” event did not actually occur, the percentage of cases where the assumption The TN at the intersection of the "Normal" row and the "Network Congestion" column indicates the percentage of cases where a "Network Congestion" event was incorrectly assumed to have occurred when a "Normal" event occurred that was not the assumed "Inadequate Communication Quality" (i.e., the percentage of cases where a "Inadequate Communication Quality" event did not actually occur but was correctly assumed to have occurred ...)).

FN at the intersection of the row "inadequate communication quality" and the column "normal" indicates the percentage of cases where the occurrence of the assumed "inadequate communication quality" event was incorrectly estimated as a different "normal" event when the assumed "inadequate communication quality" event occurred, TP at the intersection of the row "inadequate communication quality" and the column "inadequate communication quality" indicates the percentage of cases where the occurrence of the assumed "inadequate communication quality" event was correctly estimated as a "inadequate communication quality" event when the assumed "inadequate communication quality" event occurred, and FN at the intersection of the row "inadequate communication quality" and the column "network congestion" indicates the percentage of cases where the occurrence of the assumed "inadequate communication quality" event was incorrectly estimated as a different "network congestion" event when the assumed "inadequate communication quality" event occurred.

Furthermore, the TN at the intersection of the "Network Congestion" row and the "Normal" column indicates the percentage of cases where a "normal" event was estimated to have occurred when a "network congestion" event occurred that was not the assumed "inadequate communication quality" (i.e., when the assumed "inadequate communication quality" event did not actually occur, the percentage of cases where the fact (the assumed "inadequate communication quality" event did not occur) was correctly estimated), the FP at the intersection of the "Network Congestion" row and the "inadequate communication quality" column indicates the percentage of cases where a "network congestion" event occurred that was not the assumed "inadequate communication quality" and the percentage of cases where the incorrectly estimated fact (the assumed "inadequate communication quality" event did not occur) was correctly estimated), the TN at the intersection of the "Network Congestion" row and the "Network Congestion" column indicates the percentage of cases where a "network congestion" event occurred that was not the assumed "inadequate communication quality" and the percentage of cases where the "network congestion" event occurred that was not the assumed "inadequate communication quality" (i.e., when the assumed "inadequate communication quality" event did not actually occur, the percentage of cases where the correct estimation (the assumed "inadequate communication quality" event did not occur) was correctly estimated).

In FIG. 31(c), the TN at the intersection of the "Normal" row and the "Normal" column indicates the percentage of cases where a "normal" event other than the assumed "network congestion" occurred and the assumption that the "normal" event occurred was assumed (i.e., when the assumed "network congestion" event did not actually occur, the assumption (the assumption that the "network congestion" event did not occur) was correctly assumed), the TN at the intersection of the "Normal" row and the "Inadequate communication quality" column indicates the percentage of cases where a "normal" event other than the assumed "network congestion" occurred and the assumption (the assumption that the "network congestion" event did not occur) was correctly assumed), and the FP at the intersection of the "Normal" row and the "Network congestion" column indicates the percentage of cases where a "normal" event other than the assumed "network congestion" occurred and the assumption (the assumption that the "network congestion" event did not actually occur) was incorrectly assumed to have occurred.

In addition, the TN at the intersection of the row of "inadequate communication quality" and the column of "normal" indicates the percentage of cases where a "normal" event was assumed to have occurred when an "inadequate communication quality" event occurred that was not the assumed "network congestion" (i.e., when the assumed "network congestion" event did not actually occur, the percentage of cases where the event was correctly assumed to have occurred (i.e., the assumed "network congestion" event did not occur)). In addition, the TN at the intersection of the row of "inadequate communication quality" and the column of "inadequate communication quality" indicates the percentage of cases where a "normal" event was assumed to have occurred when an "inadequate communication quality" event occurred that was not the assumed "network congestion" (i.e., when the assumed The FP at the intersection of the "inadequate communication quality" row and the "network congestion" column indicates the percentage of cases where an "inadequate communication quality" event, which is not the assumed "network congestion", was correctly estimated to have occurred (i.e., when the assumed "network congestion" event did not actually occur, the percentage of cases where the FP correctly estimated that the assumed "network congestion" event had occurred) when an "inadequate communication quality" event, which is not the assumed "network congestion" event, occurred (i.e., when the assumed "network congestion" event did not actually occur), the percentage of cases where the FP correctly estimated that the assumed "network congestion" event had occurred.

Furthermore, the FN at the intersection of the "Network Congestion" row and the "Normal" column indicates the percentage of cases where the occurrence of the assumed "Network Congestion" event was mistakenly estimated as a different "Normal" event when the assumed "Network Congestion" event occurred, the FN at the intersection of the "Network Congestion" row and the "Poor Communication Quality" column indicates the percentage of cases where the occurrence of the assumed "Network Congestion" event was mistakenly estimated as a different "Poor Communication Quality" event when the assumed "Network Congestion" event occurred, and the TP at the intersection of the "Network Congestion" row and the "Network Congestion" column indicates the percentage of cases where the occurrence of the assumed "Network Congestion" event was correctly estimated as a "Network Congestion" event when the assumed "Network Congestion" event occurred.

The method of generating a confusion matrix will be explained. Figure 32 is a diagram for explaining the method of generating a confusion matrix. Note that in Figure 32, the method of generating a confusion matrix will be explained using as an example the case of generating a confusion matrix when the result of the network scan shown in Figure 31 (a) is "normal".

Referring to FIG. 32, since the result of the network scan is "normal," in the column direction, "normal" is an expected event, while "poor communication quality" and "network congestion" are unexpected events.

The estimation means 16 trains the estimation algorithm ALG using old training data consisting of initial sample data D_sample_initial, and then estimates the results of the network scan using the estimation algorithm ALG using a validation set (= consisting of one row of the subsets V _1' to V _8' shown in FIG. 29 ). Based on the estimated network scan results, the estimation means 16 executes a process of counting A, which is the number of correctly estimated expected events (= "normal") when an expected event (= "normal") occurs, B, which is the number of erroneously estimated expected events (= "normal") as unexpected events (= "poor communication quality") when an expected event (= "normal") occurs, and C, which is the number of erroneously estimated expected events (= "normal") as unexpected events (= "network congestion") when an expected event (= "normal") occurs, for all rows of the subsets V _1' to V _8' shown in FIG. 29 .

₂₉ , there are eight pairs {RTT _m' , CMbest _m' } of scan response delay RTT _m' and communication characteristic CMbest m _' , so if the expected event (="normal" ₎ is correctly estimated, A = 8 and B = C = 0 are obtained. As a result, when counting A, B, and C is performed for all rows of subsets V _1' to V ₈ ' shown in FIG. 29, if the expected event (="normal") is correctly estimated in each subset, A = 64 and B = C = 0 are obtained.

The estimation means 16 uses the validation set (= consisting of one row of the subsets V1 _' to V8 _' shown in FIG. 29) to execute a process of estimating the results of the network scan by the estimation algorithm ALG for all rows of the subsets V1 _' to V8 _' shown in FIG. 29, counts A, B, and C, and assigns the counted A, B, and C to a confusion matrix table as shown in FIG. 32.

Then, the estimation means 16 obtains A as OE, B+C as OIE, and A+B+C as ECE.

Here, OE represents the total number of events that occurred as expected but were correctly estimated, ECE represents the total number of events that occurred as expected, and OIE represents the total number of events that occurred as expected but were incorrectly estimated as unexpected.

The estimation means 16 then calculates TP and FN based on the acquired OE, OIE, and ECE using the following formula.

By calculating TP and FN based on OE, OIE, and ECE using equation (14), TP and FN in the first row of the confusion matrix CFM shown in FIG. 31(a) are obtained. In this case, as shown in FIG. 31(a), FN is obtained as the combined value of the two elements "inadequate communication quality" and "network congestion" in the confusion matrix. In this way, TP and FN are obtained based on the estimated results of the network scan.

Then, the estimation means 16 calculates FP and TN based on the calculated TP and FN using the following formula.

The TP at the intersection of the "Normal" row and the "Normal" column shown in FIG. 31(a) is calculated by substituting the total number of correctly estimated "normal" events that occurred as expected, OE, and the total number of "normal" events that occurred as expected, ECE, into formula (14A).The FP at the intersection of the "Insufficient communication quality" row and the "Normal" column shown in FIG. 31(a) and the FP at the intersection of the "Network congestion" row and the "Normal" column are calculated by substituting the TP calculated by formula (14A) into formula (15A).

Furthermore, FN (=FN1) at the intersection of the "Normal" row and the "Insufficient communication quality" column shown in FIG. 31(a) is calculated by substituting the total number OIE of "Normal" events that occurred as expected but were erroneously estimated as "Insufficient communication quality" events that occurred unexpectedly, and the total number ECE of "Normal" events that occurred as expected, into formula (14B). Then, TN at the intersection of the "Insufficient communication quality" row and the "Insufficient communication quality" column shown in FIG. 31(a) and TN at the intersection of the "Network congestion" row and the "Insufficient communication quality" column are calculated by substituting FN (=FN1) calculated by formula (14B) into formula (15B).

Furthermore, FN (=FN2) at the intersection of the "Normal" row and the "Network Congestion" column shown in FIG. 31(a) is calculated by substituting the total number OIE of "Normal" events that occurred as expected but were erroneously estimated as "Network Congestion" events that occurred unexpectedly, and the total number ECE of "Normal" events that occurred as expected, into formula (14B). Then, TN at the intersection of the "Insufficient Communication Quality" row and the "Network Congestion" column shown in FIG. 31(a) and TN at the intersection of the "Network Congestion" row and the "Network Congestion" column are calculated by substituting FN (=FN2) calculated by formula (14B) into formula (15B).

When the estimation means 16 calculates TP, FN, TN, and FP using the method described above, it generates a confusion matrix CFM shown in FIG. 31(a) consisting of the calculated TP, FN, TN, and FP.

The estimation means 16 generates the confusion matrix CFM shown in (b) and (c) of FIG. 31 in a similar manner. In this case, since the result of the network scan is "inadequate communication quality" in the confusion matrix CFM shown in (b) of FIG. 31, in the column direction, "inadequate communication quality" is an expected event, and "normal" and "network congestion" are unexpected events. As a result, by the above-mentioned method, TP and FN in the second row of the confusion matrix CFM shown in (b) of FIG. 31 are calculated based on OE, OIE, and ECE, and then FP and TN in the first and third rows are calculated based on TP and FN.

In addition, in the confusion matrix CFM shown in FIG. 31(c), since the result of the network scan is "network congestion", in the column direction, "network congestion" is an expected event, and "normal" and "poor communication quality" are unexpected events. As a result, by the above-mentioned method, TP and FN in the third row of the confusion matrix CFM shown in FIG. 31(c) are calculated based on OE, OIE, and ECE, and then FP and TN in the first and second rows are calculated based on TP and FN.

In this way, the estimation means 16 acquires TP and FN from the estimation results of the network scan, calculates FP and TN based on the acquired TP and FN, and generates a confusion matrix CFM consisting of TP, FN, FP, and TN. In addition, the estimation means 16 calculates FN, FP, and TN by adding up multiple elements in the confusion matrix.

This method of generating the confusion matrix CFM differs from the general method of generating a confusion matrix, in which TP, FN, FP, and TN are obtained based on the observation of the subject for which the confusion matrix is to be generated, and a confusion matrix is generated.

As described above, the estimation means 16 trains the estimation algorithm ALG using old training data consisting of the initial sample data D_sample_initial, and then generates a confusion matrix CFM when estimating the results of the network scan by the estimation algorithm ALG using the validation set (= consisting of all rows of the subsets V _{1 '} to V _{8 '} shown in FIG. 29 ).

In addition, the estimation means 16 trains the estimation algorithm ALG using the tentative sample data D_sample_tentative_u, and then similarly estimates the results of the network scan using the estimation algorithm ALG using a validation set (= consisting of all rows of the subsets V1 _' to V8 _' shown in Figure 29), and generates a confusion matrix nCFM when the results of the network scan are estimated in the same manner as the confusion matrix CFM.

Furthermore, after generating the confusion matrix nCFM, the estimation means 16 estimates the result of the network scan by the estimation algorithm ALG using the test data (=consisting of all of the subsets TD ₁ to TD ₄ shown in FIG. 30 ), and generates a confusion matrix tCFM when the result of the network scan is estimated by the same method as the confusion matrix CFM.

In this case, the estimation means 16 estimates the results of the network scan using the test data (= consisting of one row of the subsets TD ₁ to TD ₄ shown in FIG. 30) with the estimation algorithm ALG, and executes the process of counting A, B, and C for all rows of the subsets TD ₁ to TD ₄ shown in FIG. 30 based on the estimated results of the network scan.

In one row of subsets TD ₁ to TD ₄ shown in Fig. 30, there are four pairs {RTT _m' , CMbest _m' } of scan response delay RTT _m' and communication characteristic CMbest m _' , so if the expected event (="normal") is correctly estimated, A = 4 and B = C = 0 are obtained. As a result, when counting A, B, and C is performed for all rows of subsets TD ₁ to TD ₄ shown in Fig. 30, if the expected event (="normal") is correctly estimated in all subsets, A = 32 and B = C = 0 are obtained. The same applies when the expected event is "poor communication quality" or "network congestion".

When the estimation means 16 generates the confusion matrices CFM, nCFM, and tCFM, it outputs the generated confusion matrices CFM, nCFM, and tCFM to the calculation means 15.

The calculation means 15 receives the confusion matrices CFM, nCFM, and tCFM from the estimation means 16. Then, the calculation means 15 uses the elements TP, TN, FP, and FN of the confusion matrices CFM, nCFM, and tCFM to calculate Precision, which is the ratio of the proportion actually classified as TP among the proportion classified as TP or FP, and Recall, which is the ratio of the proportion actually classified as TP among the proportion classified as TP or FN.

Precision is calculated using equation (16), and Recall is calculated using equation (17).

Then, after calculating Precision and Recall, the calculation means 15 calculates the F1 score _F1 by substituting the calculated Precision and Recall into the following equation.

The calculation means 15 calculates an F1 score F ₁ by equations (16) to (18) based on the TP, TN, FP, and FN of the confusion matrix CFM. The calculation means 15 also calculates an F1 score nF ₁ _u by equations (16) to (18) based on the TP, TN, FP, and FN of the confusion matrix nCFM. The calculation means 15 also calculates an F1 score tF ₁ by equations (16) to (18) based on the TP, TN, FP, and FN of the confusion matrix tCFM. The calculation means 15 then outputs the F1 scores F ₁ , nF ₁ _u, and tF ₁ to the estimation means 16.

The F1 score _F1 is an evaluation index EVD that represents the accuracy of the estimation result when the estimation algorithm ALG is trained using the initial sample data D_sample_initial and the network scan result is estimated by the estimation algorithm ALG (the estimation algorithm ALG after training) using the validation set (= consisting of all rows of the subsets V1 _' to V8 _' shown in Figure 29).

In addition, the F1 score nF ₁ _u is an evaluation index nEVD_u that represents the accuracy of the estimation result when the estimation algorithm ALG is trained using the tentative sample data D_sample_tentative_u and the network scan result is estimated by the estimation algorithm ALG (the estimation algorithm ALG after training) using the validation set (= consisting of all rows of the subsets V _1' to _V 8' shown in Figure 29).

Furthermore, the F1 score _tF1 is an evaluation index tEVD that represents the accuracy of the estimation result when the network scan result is estimated by the estimation algorithm ALG (the estimation algorithm ALG after training) using the test data (= consisting of all _{rows of} the subsets TD1 to TD4 shown in Figure 30).

In an embodiment of the present invention, the Matthews correlation coefficients MCC, nMCC_u, and tMCC may be used as the evaluation indices EVD, nEVD_u, and tEVD, respectively, instead of the F1 scores F ₁ , nF ₁ _u, and tF _1. In this case, the calculation means 15 calculates the Matthews correlation coefficient MCC by equation (19) based on the TP, TN, FP, and FN of the confusion matrix CFM, calculates the Matthews correlation coefficient nMCC_u by equation (19) based on the TP, TN, FP, and FN of the confusion matrix nCFM, and calculates the Matthews correlation coefficient tMCC by equation (19) based on the TP, TN, FP, and FN of the confusion matrix tCFM.

The Matthews correlation coefficient MCC calculated by formula (19) is "+1" when the best training data is used, "-1" when the worst training data is used, and "0" when neutral (neither "best" nor "worst") training data is used.

When the calculation means 15 calculates the Matthews correlation coefficients MCC, nMCC_u, and tMCC, it outputs the Matthews correlation coefficient MCC to the estimation means 16 as an evaluation index EVD, outputs the Matthews correlation coefficient nMCC_u to the estimation means 16 as an evaluation index nEVD_u, and outputs the Matthews correlation coefficient tMCC to the estimation means 16 as an evaluation index tEVD.

Upon receiving the evaluation index nEVD_u consisting of the F1 score nF ₁ _1 and the evaluation index EVD consisting of the F1 score F ₁ from the calculation means 15, the estimation means 16 determines whether the F1 score nF ₁ _1 is greater than the F1 score F ₁ .

When the estimation means 16 determines that the F1 score nF ₁ _1 is greater than the F1 score F ₁ , the estimation means 16 sets the F1 score nF ₁ _1 to the F1 score F ₁ , and sets the tentative sample data D_sample_tentative_1 including the subset d ₁ when the F1 score nF ₁ _1 is obtained as the training data. Then, the estimation means 16 replaces the variables of the estimation algorithm ALG using the training data by a method described later. After that, the estimation means 16 trains the estimation algorithm ALG using the next subset d ₂ , estimates the result of the network scan by the estimation algorithm ALG (the estimation algorithm ALG after training) using the validation set (=consisting of all of _the subsets _V 1' to V 8'), and generates a confusion matrix nCFM_2 when the result of the network scan is estimated. Then, the estimation means 16 outputs the generated confusion matrix nCFM_2 to the calculation means 15.

On the other hand, when the estimation means 16 determines that the F1 score nF ₁ _1 is not greater than the F1 score F ₁ , it does not set the F1 score nF ₁ _1 to the F1 score F ₁ , but trains the estimation algorithm ALG using tentative sample data D_sample_tentative_2 including the next subset d ₂ , estimates the result of the network scan with the estimation algorithm ALG (the estimation algorithm ALG after training) using the validation set (=consisting of all of the subsets V ₁ ' to _{V 8'} ), generates a confusion matrix nCFM_2 when the result of the network scan is estimated, and outputs the confusion matrix nCFM_2 to the calculation means 15.

Thereafter, when the estimation means 16 receives the evaluation index nEVD_u consisting of the F1 score nF ₁ _2 from the calculation means 15, the estimation means 16 determines whether or not the F1 score nF ₁ _2 is greater than the F1 score F ₁ .

When the estimation means 16 determines that the F1 score nF ₁ _2 is greater than the F1 score F ₁ , it sets the F1 score nF ₁ _2 to the F1 score F ₁ , and sets the tentative sample data D_sample_tentative_2 including the subset d ₂ when the F1 score nF ₁ _2 is obtained as the training data. Then, the estimation means 16 replaces the variables of the estimation algorithm ALG using the training data by a method described later. After that, the estimation means 16 trains the estimation algorithm ALG using the tentative sample data D_sample_tentative_3 including the next subset d ₃ , estimates the result of the network scan by the estimation algorithm ALG (the estimation algorithm ALG after training) using the validation set (=consisting of all of the subsets V _1' to V _8' ), and generates a confusion matrix nCFM_3 when the result of the network scan is estimated. Then, the estimation means 16 outputs the generated confusion matrix nCFM_3 to the calculation means 15.

On the other hand, when the estimation means 16 determines that the F1 score nF ₁ _2 is not greater than the F1 score F ₁ , it does not set the F1 score nF ₁ _2 to the F1 score F ₁ , but trains the estimation algorithm ALG using tentative sample data D_sample_tentative_3 including the next subset d ₃ , estimates the result of the network scan by the estimation algorithm ALG (the estimation algorithm ALG after training) using the validation set (=consisting of all of the subsets _V 1' to _{V 8'} ), and generates a confusion matrix nCFM_3 when the result of the network scan is estimated. Then, the estimation means 16 outputs the generated confusion matrix nCFM_3 to the calculation means 15.

Thereafter, the estimation means 16 repeatedly executes the above-mentioned operations for all of the subsets d ₃ to d _U using the F1 scores F ₁ and nF ₁ _u.

In addition, when the estimation means 16 receives an evaluation index EVD consisting of the Matthews correlation coefficient MCC and an evaluation index nEVD_1 consisting of the Matthews correlation coefficient nMCC_1 from the calculation means 15, it determines whether the Matthews correlation coefficient nMCC_1 is greater than the Matthews correlation coefficient MCC.

When the estimation means 16 determines that the Matthews correlation coefficient nMCC_1 is greater than the Matthews correlation coefficient MCC, it sets the Matthews correlation coefficient nMCC_1 to the Matthews correlation coefficient MCC, and sets the tentative sample data D_sample_tentative_1 including the subset d ₁ when the Matthews correlation coefficient nMCC_1 is obtained as the training data. Then, the estimation means 16 replaces the variables of the estimation algorithm ALG using the training data by a method described later. After that, the estimation means 16 trains the estimation algorithm ALG using tentative sample data D_sample_tentative_2 including the next subset d ₂ , estimates the result of the network scan by the estimation algorithm ALG using the validation set (=consisting of all of the subsets V _1' to V _8' ), and generates a confusion matrix nCFM_2 when the result of the network scan is estimated. Then, the estimation means 16 outputs the generated confusion matrix nCFM_2 to the calculation means 15.

On the other hand, when the estimation means 16 determines that the Matthews correlation coefficient nMCC_1 is not greater than the Matthews correlation coefficient MCC, it does not set the Matthews correlation coefficient nMCC_1 to the Matthews correlation coefficient MCC, but trains the estimation algorithm ALG using the tentative sample data D_sample_tentative_2 including the next subset _d2 , estimates the result of the network scan by the estimation algorithm ALG using the validation set (=consisting of all of the subsets V1 _' to V8 _' ), and generates a confusion matrix nCFM_2 when the result of the network scan is estimated. Then, the estimation means 16 outputs the generated confusion matrix nCFM_2 to the calculation means 15.

Then, when the estimation means 16 receives the evaluation index nEVD_2 consisting of the Matthews correlation coefficient nMCC_2 from the calculation means 15, it determines whether the Matthews correlation coefficient nMCC_2 is greater than the Matthews correlation coefficient MCC.

When the estimation means 16 determines that the Matthews correlation coefficient nMCC_2 is greater than the Matthews correlation coefficient MCC, it sets the Matthews correlation coefficient nMCC_2 to the Matthews correlation coefficient MCC, and sets the tentative sample data D_sample_tentative_2 including the subset d ₂ when the Matthews correlation coefficient nMCC_2 is obtained as the training data. Then, the estimation means 16 replaces the variables of the estimation algorithm ALG using the training data by a method described later. After that, the estimation means 16 trains the estimation algorithm ALG using tentative sample data D_sample_tentative_3 including the next subset d ₃ , estimates the result of the network scan by the estimation algorithm ALG using the validation set (=consisting of all of the subsets V _1' to V _8' ), and generates a confusion matrix nCFM_3 when the result of the network scan is estimated. Then, the estimation means 16 outputs the generated confusion matrix nCFM_3 to the calculation means 15.

On the other hand, when the estimation means 16 determines that the Matthews correlation coefficient nMCC_2 is not greater than the Matthews correlation coefficient MCC, it does not set the Matthews correlation coefficient nMCC_2 to the Matthews correlation coefficient MCC, but trains the estimation algorithm ALG using the tentative sample data D_sample_tentative_3 including the next subset _d3 , estimates the result of the network scan by the estimation algorithm ALG using the validation set (=consisting of all of the subsets V1 _' to V8 _' ), and generates a confusion matrix nCFM_3 when the result of the network scan is estimated. Then, the estimation means 16 outputs the generated confusion matrix nCFM_3 to the calculation means 15.

Thereafter, the estimation means 16 repeatedly executes the above-mentioned operations for all of the subsets d ₃ to d _U using the Matthews correlation coefficients MCC and nMCC_u.

[Estimated network scan results]
In training the above-mentioned estimation algorithm ALG, the estimation means 16 uses the initial sample data D_sample_initial and tentative sample data D_sample_tentative_1 to D_sample_tentative_U each including a subset d ₁ to d _U generated from the standard sample data D_similarity to estimate the result of the network scan by the following method.

In addition, in explaining the method of estimating the results of a network scan, the scan response delays {RTT _1' _known}_z to {RTT _M' _known}_z contained in each of the hypothetical sample data D_sample_tentative_1 to D_sample_tentative_U are denoted as "scan response delays RTT _m" , and the communication characteristic CMbest _m _unknown contained in each of the hypothetical sample data D_sample_tentative_1 to D_sample_tentative_U and the communication characteristics {CMbest ₁ ' _known}_z to {CMbest _M' _known}_z contained in the standard sample data D_similarity are denoted as "communication characteristic CMbest".

The estimation means 16 receives the scan response delay RTT _m and the communication characteristic CMbest from the extraction means 14. Then, the estimation means 16 classifies the terminal devices having the communication characteristic CMbest into G groups CLS ₁ to CLS _G (G is an integer equal to or greater than 3) using an LDA (Linear Discriminant Analysis) algorithm.

The LDA algorithm classifies terminal devices having communication characteristics CMbest into G groups CLS ₁ to CLS _G so as to maximize the ratio of between-group variance and within-group variance, thus ensuring maximum separation between the groups CLS ₁ to CLS _G from each other.

The LDA algorithm spheres the data (terminal devices with communication characteristics CMbest) based on the common covariance and classifies the data (terminal devices with communication characteristics CMbest) into the closest group centroid in the transformed space.

To achieve this, the separability between different groups (also called "intergroup covariance"), which is the distance between the means of different groups, is calculated. Then, the distance between the mean and the sample of each group (also called "intragroup covariance") is calculated. This allows a low-dimensional space to be constructed that maximizes the intergroup covariance and minimizes the intragroup covariance. Then, in the constructed low-dimensional space, data (terminal devices having communication characteristics CMbest) are classified into the closest group centroid. The LDA algorithm is described in "T. Hastie, R. Tibshirani, and J. H. Friedman, "The elements of statistical learning: data mining, inference, and prediction," 2nd ed. Chapter 4. pp.106-118, New York: Springer, 2009." and is publicly known. Therefore, a detailed classification method using the LDA algorithm will be omitted.

Moreover, the number G of the groups CLS ₁ to CLS _G is set to the same number as the number of causes of the network scan results attributable to the communication channel with the terminal device among the estimated results of the network scan. That is, since the three results of "normal", "inadequate communication quality" and "network congestion" are causes attributable to the communication channel with the terminal device, the number G of the groups CLS ₁ to CLS _G is set to "3". As a result, the estimation means 16 classifies the terminal devices having the communication characteristic CMbest into three groups CLS ₁ to CLS ₃ using the LDA algorithm. Then, the group CLS ₁ is a "normal" group, the group CLS ₂ is a "inadequate communication quality" group, and the group CLS ₃ is a "network congestion" group.

When classifying terminal devices having communication characteristics CMbest into groups _CLS1 to _CLS3 , the estimation means 16 learns an LDA algorithm using scan data when the network scan is "network congested" (communication characteristics CMbest when "network is congested"), and classifies terminal devices having communication characteristics CMbest into three groups _CLS1 to _CLS3 using the learned LDA algorithm. This makes it possible to most accurately estimate that "network congestion" is the cause of the failure of the network scan.

The estimation means 16 uses Model1, Model2 and Model3 for estimating the results of a network scan. Model1 handles groups _CLS1 and _CLS2 , Model2 handles groups _CLS1 and _CLS3 , and Model3 handles groups _CLS2 and _CLS3 .

Then, when Model 1 is used, the estimation means 16 calculates a D score D_1 that discriminates between groups CLS ₁ and CLS ₂ , when Model 2 is used, calculates a D score D_2 that discriminates between groups CLS ₁ and CLS ₃ , and when Model 3 is used, calculates a D score D_3 that discriminates between groups CLS ₂ and CLS ₃ .

When the D scores D_1 to D_3 are calculated, a discriminant function δ _pq (x) expressed by the following equation is used.

A vector consisting of the scan response delay RTT _m and the communication characteristic CMbest is substituted for x in formula (20A). Then, x consists of the scan response delay RTT _m and the communication characteristic CMbest (optimum training data) when the result of the network scan is known, or the scan response delay RTT _m and the communication characteristic CMbest when the result of the network scan is unknown. Furthermore, the average μ _p of the communication characteristic shown in formula (20C) is substituted for μ _g in formula (20A) when calculating the discriminant function of group p, and the average μ q of the communication characteristic shown in formula (20D) is substituted for μ _g in formula (20A) when calculating the discriminant function of group q. Furthermore, the prior average Π _p of the communication characteristic shown in formula (20E) is substituted for the prior average Π _g in formula (20A) when calculating the discriminant function of group p, and the prior average Π q of the communication characteristic shown in formula (20F) is substituted for the prior average Π _g in formula (20A). The joint covariance matrix C _pq of formula (20A) is represented by formula (20B). On the right side of formula (20B), the first term is a covariance matrix in group p, and the second term is a covariance matrix in group q. Then, the communication characteristics classified into group p (a vector consisting of scan response delay RTT _m and communication characteristics CMbest) are substituted into x _mp of formula (20B), and the communication characteristics classified into group q (a vector consisting of scan response delay RTT _m and communication characteristics CMbest) are substituted into x _mq of formula (20B).

In formulas (20B), (20C), and (20D), M _p is the total number of communication characteristics (vectors consisting of scan response delay RTT _m and communication characteristics CMbest) classified into group p, M _q is the total number of communication characteristics (vectors consisting of scan response delay RTT _m and communication characteristics CMbest) classified into group q, m _p is an argument indicating one communication characteristic (vectors consisting of scan response delay RTT _m and communication characteristics CMbest) classified into group p, and m _q is an argument indicating one communication characteristic (vectors consisting of scan response delay RTT _m and communication characteristics CMbest) classified into group q. In formulas (20E) and (20F), M is the total number of communication characteristics (vectors consisting of scan response delay RTT _m and communication characteristics CMbestt). Formula (20C) represents the average μ _p of the communication characteristics (vector consisting of scan response delay RTT _m and communication characteristic CMbest) classified into group p, formula (20D) represents the average μ _q of the communication characteristics (vector consisting of scan response delay RTT _m and communication characteristic CMbest) classified into group q, formula (20E) represents the prior average Π _p of the communication characteristics (vector consisting of scan response delay RTT _m and communication characteristic CMbest) classified into group p, and formula (20F) represents the prior average Π _q of the communication characteristics (vector consisting of scan response delay RTT _m and communication characteristic CMbest) classified into group q.

Equation (20B) represents a merged covariance matrix C pq obtained by merging two covariance matrices in two groups (two groups CLS ₁ and CLS ₂ ; two groups CLS ₁ and CLS ₃ ; or two groups CLS ₂ and CLS ₃ ). When calculating a merged covariance matrix C ₁₂ obtained by merging two covariance matrices in two groups CLS ₁ and CLS ₂ , in equation (20A), p represents group CLS ₁ , and q represents group CLS _2. Furthermore, when calculating a merged covariance matrix _{C 13 obtained} by merging two covariance matrices in two groups CLS ₁ and CLS ₃ , in equation (20A), p represents group CLS ₁ , and q represents group CLS ₃ . Furthermore, when calculating a merged covariance matrix _C23 by merging the two covariance matrices in the two groups _CLS2 and _CLS3 , p represents group _CLS2 and q represents group _CLS3 in formula (20A).

The joint covariance matrix C ₁₂ in Model 1 is expressed by the following equation.

In formula (21), _M1 corresponds to _Mp in formula (20B), _M2 corresponds to _Mq in formula (20B), _m1 corresponds to _mp in formula (20B), and _m2 corresponds to _mq in formula (20B). Also, _μ1 is calculated by substituting _M1 for _Mp in formula (20C) and substituting the communication characteristic classified into group _CLS1 (a vector consisting of scan response delay RTT _m and communication characteristic CMbest) into _xmp in formula (20C), and _μ2 is calculated by substituting _M2 for _Mq in formula (20D) and substituting the communication characteristic classified into group _CLS2 (a vector consisting of scan response delay RTT _m and communication characteristic CMbest) into _xmq in formula (20D). The joint covariance matrix _C12 is represented by the sum of the covariance matrix in group _CLS1 and the covariance matrix in group _CLS2 .

The joint covariance matrix C ₁₃ in Model 2 is expressed by the following equation.

In formula (22), _M1 corresponds to _Mp in formula (20B), _{M3 corresponds to Mq in formula (20B), m1 corresponds to mp} in formula (20B), and _m3 corresponds to _mq in formula (20B). Also, _μ1 is calculated by substituting _M1 for _Mp in formula (20C) and substituting the communication characteristic classified into group _CLS1 (a vector consisting of scan response delay RTT _m and communication characteristic CMbest) into _xmp in formula (20C), and _μ3 is calculated by substituting _M3 for _Mq in formula (20D) and substituting the communication characteristic classified into group _CLS3 (a vector consisting of scan response delay RTT _m and communication characteristic CMbest) into _xmq in formula (20D). The joint covariance matrix _C13 is represented by the sum of the covariance matrix in group _CLS1 and the covariance matrix in group _CLS3 .

The joint covariance matrix C ₂₃ in Model 3 is expressed by the following equation.

In formula (23), _M2 corresponds to _Mp in formula (20B), _M3 corresponds to _Mq in formula (20B), _m2 corresponds to _mp in formula (20B), and _m3 corresponds to _mq in formula (20B). _μ2 is calculated by substituting _M2 for _Mp in formula (20C) and substituting the communication characteristic classified into group _CLS2 (a vector consisting of scan response delay RTT _m and communication characteristic CMbest) into _xmp in formula (20C), and _μ3 is calculated by substituting _M3 for _Mq in formula (20D) and substituting the communication characteristic classified into group _CLS3 (a vector consisting of scan response delay RTT _m and communication characteristic CMbest) into _xmq in formula (20D). The joint covariance matrix _C23 is represented by the sum of the covariance matrix in group _CLS2 and the covariance matrix in group _CLS3 .

When calculating the D score D_1 in Model 1, the discriminant function ^f1 in group CLS ₁ and the discriminant function ^f2 in group CLS ₂ are expressed by the following equations.

In formula (24A), the prior average Π ₁ is calculated by substituting M ₁ into M _p in formula (20E) and substituting the total number of communication characteristics (vectors consisting of scan response delay RTT _m and communication characteristics CMbest) into M in formula (20E). In formula (24B), the prior average Π ₂ is calculated by substituting M ₂ into M _q in formula (20F) and substituting the total number of communication characteristics (vectors consisting of scan response delay RTT _m and communication characteristics CMbest) into M in formula (20F). In addition, the average μ ₁ in formula (24A) and the average μ ₂ in formula (24B) are calculated by the method described in formula (21).

Furthermore, when calculating the D score D_2 in Model 2, the discriminant function ^f1 in group CLS ₁ and the discriminant function ^f2 in group CLS ₃ are expressed by the following equations.

In formula (25A), the prior average Π ₁ is calculated by substituting M ₁ into M _p in formula (20E) and substituting the total number of communication characteristics (vectors consisting of scan response delay RTT _m and communication characteristics CMbest) into M in formula (20E). In formula (25B), the prior average Π ₂ is calculated by substituting M ₃ into M _q in formula (20F) and substituting the total number of communication characteristics (vectors consisting of scan response delay RTT _m and communication characteristics CMbest) into M in formula (20F). In addition, the average μ ₁ in formula (25A) and the average μ ₃ in formula (25B) are calculated by the method described in formula (22).

Furthermore, when calculating the D score D_3 in Model 3, the discriminant function ^f1 in group CLS ₂ and the discriminant function ^f2 in group CLS ₃ are expressed by the following equations.

In formula (26A), the prior average Π ₂ is calculated by substituting M ₂ into M _p in formula (20E) and substituting the total number of communication characteristics (vectors consisting of scan response delay RTT _m and communication characteristics CMbest) into M in formula (20E). In formula (26B), the prior average Π ₃ is calculated by substituting M ₃ into M _q in formula (20F) and substituting the total number of communication characteristics (vectors consisting of scan response delay RTT _m and communication characteristics CMbest) into M in formula (20F). In addition, the average μ ₂ in formula (26A) and the average μ ₃ in formula (26B) are calculated by the method described in formula (23).

The D score D_1 is expressed by the difference between the discriminant function ^f1 shown in equation (24A) and the discriminant function ^f2 shown in equation (24B), that is, ^f1 - ^f2 .

Also, the D score D_2 is represented by the difference between the discriminant function ^f1 shown in the formula (25A) and the discriminant function ^f2 shown in the formula (25B), that is, ^f1 - ^f2 .

Furthermore, the D score D_3 is represented by the difference between the discriminant function ^f1 shown in equation (26A) and the discriminant function ^f2 shown in equation (26B), that is, ^f1 - ^f2 .

Therefore, the D scores D_1 to D_3 can be calculated by using equations (20) to (26).

The estimation means 16 uses one of Model1 to Model3 to calculate the D score (one of D_1 to D_3).

Then, the estimation means 16 judges whether the D score is less than zero (=0). When the estimation means 16 judges that the D score is less than zero (=0), it increments the count value DS_count, which indicates the number of D scores less than zero (=0), by "1".

On the other hand, when the estimation means 16 determines that the D score is equal to or greater than zero (=0), it keeps the count value DS_count unchanged.

The estimation means 16 performs this process for all terminal devices that belong to one base station.

After counting the count value DS_count of the D score, the estimation means 16 sets the Model for estimating the results of the network scan to Model2 or Model3 based on the count value DS_count.

More specifically, the estimation means 16 first sets Model to "1" (i.e., Model1), and when the count value DS_count is zero (=0) or is greater than (N _terminal ×D_thres), sets the model Model to "2" (i.e., Model2). Note that N _terminal is the total number of terminal devices belonging to one base station. Also, D_thres is set to a real number in the range of 0 to 1.

Furthermore, when the count value DS_count is not zero (=0) and is equal to or less than (N _terminal ×D_thres), the estimation means 16 sets the model Model to "3" (i.e., Model 3).

Then, when the model Model is not "1" (i.e., when the model Model is "2" or "3"), the estimation means 16 determines whether the count value DS_count is zero (=0).

When it is determined that the count value DS_count is zero (=0), the estimation means 16 determines that the scan data for the terminal device is "normal."

On the other hand, when it is determined that the count value DS_count is not zero (=0), the estimation means 16 determines whether or not the count value DS_count is smaller than (N _terminal ×D_thres).

Then, when it is determined that the count value DS_count is smaller than (N _terminal ×D_thres), the estimation means 16 estimates that the cause of the failure of the network scan for a terminal device having a D score smaller than "0" is "insufficient communication quality."

On the other hand, when it is determined that the count value DS_count is not smaller than (N _terminal × D_thres) (i.e., when it is determined that the count value DS_count is greater than or equal to (N _terminal × D_thres)), the estimation means 16 estimates that the cause of the failure of the network scan for all terminal devices belonging to the base station is "network congestion" (i.e., network congestion).

In addition, when it is determined that the count value DS_count is zero (=0) and further that there has been a scan response from all ports of the terminal device, the estimation means 16 estimates that the network scan for the terminal device is "normal."

On the other hand, when it is determined that the count value DS_count is zero (=0) and further that a response has been received from some of the ports of the terminal device, the estimation means 16 estimates that the cause of the failure of the network scan for the terminal device is a "closed port."

Furthermore, when it is determined that the count value DS_count is zero (=0) and furthermore that there is no scan response from all ports of the terminal device, it is estimated that the cause of the failure of the network scan for the terminal device is "no network connection."

The estimation algorithm ALG is an algorithm that estimates the results of a network scan using the method described above.

[Variable replacement in estimation algorithm]
When the F1 score nF 1 when the result of the network scan is estimated by the estimation algorithm ALG using a validation set (=consisting of all of the subsets V _{1 ′} to V _{U ′} ) generated from the standard sample data D_similarity is larger than the F1 score F ₁ , or when the Matthews correlation coefficient _{nMCC_u} when the result of the network scan is estimated by the estimation algorithm ALG using the validation set (=consisting of all of the subsets V _{1 ′} to V _{U ′} ) is larger than the Matthews correlation coefficient MCC, the estimation means 16 replaces variables of the estimation algorithm ALG using tentative sample data D_sample_tentative_max including a subset d _max (any of the subsets d ₁ to d _{U) when the F1 score nF 1 determined to be larger than the F1 score F 1 is finally obtained, or replaces variables of the estimation algorithm ALG using a subset d max (any of the subsets d 1 to d U ) when the Matthews} correlation coefficient nMCC_max determined to be larger than the Matthews correlation coefficient MCC is finally obtained. 1. The variables of the estimation algorithm ALG are replaced with tentative sample data D_sample_tentative_max including any of the _data D_sample_tentative_max, ...

As described in the above formula (20), x in formula (20A) is a vector consisting of the scan response delay RTT _m and the communication characteristic CMbest. Furthermore, μ _p in formula (20C) is the average μ p of the communication characteristics classified into group p (vector consisting of the scan response delay RTT _m and the communication characteristic CMbest), and μ _q in formula (20D) is the average μ _q of the communication characteristics classified into group _q (vector consisting of the scan response delay RTT _m and the communication characteristic CMbest). Furthermore, the prior average Π _p in formula (20E) is the prior average of the communication characteristics classified into group p (vector consisting of the scan response delay RTT _m and the communication characteristic CMbest), and the prior average Π _q in formula (20F) is the prior average of the communication characteristics classified into group q (vector consisting of the scan response delay RTT _m and the communication characteristic CMbest).

Here, the tentative sample data D_sample_tentative_max including data of size n (any of data d ₁ to d _U ) when an F1 score nF ₁ that is ultimately determined to be greater than the F1 score F ₁ is obtained, or a subset d _max (any of subsets d ₁ to d _U ) when a Matthews correlation coefficient nMCC_u that is ultimately determined to be greater than the Matthews correlation coefficient MCC is obtained, is denoted as "sample data D_sample_optimum".

The estimation means 16 substitutes the sample data D_sample_optimum for x in the formula (20A), calculates an average of a vector consisting of the scan response delay RTT _m and communication characteristic CMbest classified into group p among the scan response delay RTT _m and communication characteristic CMbest constituting the sample data D_sample_optimum as an average μ _p , calculates an average of a vector consisting of the scan response delay RTT m and communication characteristic CMbest classified into group q among the scan response delay RTT _m and communication characteristic CMbest constituting the sample data D_sample_optimum as an average μ _q , and calculates an average of a vector consisting of the scan response delay RTT _m and communication characteristic CMbest classified into group p among the scan response delay RTT _m and communication characteristic CMbest constituting the sample data D_sample_optimum as an average μ q. The prior average of a vector consisting of scan response delays RTT _m and communication characteristics CMbest classified into group _q among the scan response delays RTT _m and communication characteristics CMbest constituting the sample data D_sample_optimum is calculated as the prior average Π _{q ,} thereby replacing the variables of the estimation algorithm ALG with the sample data D_sample_optimum.

This improves the accuracy of estimation when estimating the results of a network scan using the estimation algorithm ALG.

[Verification using test data]
After the model variables are replaced by the optimal training data, the estimation means 16 uses the test data (=consisting of all of the subsets TD ₁ to TD ₄ ) to estimate the results of the network scan with the model (estimation algorithm ALG) in which the model variables have been replaced, and generates a confusion matrix tCFM based on the estimated results of the network scan and outputs it to the calculation means 15.

The calculation means 15 receives the confusion matrix tCFM from the estimation means 16, calculates the F1 score _tF1 based on the elements TP, TN, FP, and FN of the received confusion matrix tCFM using equations (16) to (18), and outputs the calculated F1 score _tF1 to the estimation means 16.

The estimation means 16 receives the F1 score _tF1 from the calculation means 15 and determines whether the F1 score _tF1 is equal to or greater than the F1 score _F1 .

When the estimation means 16 determines that the F1 score tF ₁ is equal to or greater than the F1 score F ₁ , it approves the model (estimation algorithm ALG) in which the model variables have been replaced and the training data (optimal training data).

When it is determined that the F1 score _tF1 is not equal to or greater than the F1 score _F1 , the estimation means 16 does not approve the model (estimation algorithm ALG) and training data (optimum training data) in which the model variables have been replaced.

In this way, the estimation means 16 uses the test data (=consisting of all of the subsets TD ₁ to TD ₄ ) to estimate the results of a network scan using the model (estimation algorithm ALG) in which the model variables have been replaced, and verifies the model (estimation algorithm ALG) in which the model variables have been replaced and the training data (optimum training data) by confirming that the F1 score tF ₁ is equal to or greater than the F1 score F _1. That is, the estimation means 16 uses the test data (=consisting of all of the subsets TD ₁ to TD ₄ ) to verify that the model (estimation algorithm ALG) in which the model variables have been replaced can more accurately estimate the results of a network scan performed in a communication environment in which the results of the network scan are unknown, compared to the model (estimation algorithm ALG) before the model variables were replaced.

33 is a flow chart for explaining the operation of a network scan. When the operation of a network scan is started, the network scanner 11 obtains a scan list including addresses Add ₁ to Add _M and port numbers PN _r of terminal devices to be the targets of the network scan, and a scan schedule from the scan scheduler 17 (step S1).

Then, the network scanner 11 sets m=1 (step S2), and sets PN _r =1 (step S3).

Then, the network scanner 11 performs a network scan with or without monitoring background traffic (step S4). Then, the network scanner 11 obtains a scan response and a latency (step S5).

Then, the network scanner 11 determines whether PN _r =PN _r _MAX (the maximum value of the port number PN _r ) is satisfied (step S6).

When it is determined in step S6 that _PNr is not equal to _{PNr_MAX} , the network scanner 11 sets _PNr to _PNr +1 (step S7). After that, the series of operations proceeds to step S4, and steps S4 to S7 are repeatedly executed until it is determined in step S6 that _PNr is equal to _{PNr_MAX} .

Then, if it is determined in step S6 that PN _r =PN _r _MAX, the network scanner 11 determines whether or not m =M (step S8).

If it is determined in step S8 that m is not equal to M, the network scanner 11 sets m to m+1 (step S9). After that, the series of operations proceeds to step S3, and steps S3 to S9 are repeatedly executed until it is determined in step S8 that m is equal to M.

Then, in step S8, if it is determined that m=M, the network scanner 11 extracts communication characteristics (multiple communication characteristics such as scan response delay RTT _m ) from the scan response (step S10), and outputs the extracted communication characteristics and the scan response to the scan analysis manager 12.

The scan analysis manager 12 receives scan responses and communication characteristics from the network scanner 11, and creates or updates the host information database 13 based on the received scan responses and communication characteristics.

Then, the estimation means 16 trains the estimation algorithm ALG that estimates the results of the network scan, and selects training data (step S11). In this case, the estimation means 16 estimates the results of the network scan using the standard sample data D_similarity, and trains the estimation algorithm ALG by evaluating the accuracy of the estimated results. In addition, the estimation means 16 selects the sample data (sample data D_sample_optimum) when the accuracy of the estimated results is the highest as the training data.

In step S11, the scan scheduler 17 receives from the estimation means 16 the estimated results of the network scan when accuracy is highest, creates a rescan list based on the received estimated results of the network scan (step S12), and outputs the created rescan list to the network scanner 11.

The reason for creating a rescan list is that if the result of a network scan is "insufficient communication quality" or "network congestion," running a network scan at a different time may result in the network scan being "normal."

After step S12, the network scanner 11 determines whether the rescan list is empty (step S13).

When it is determined in step S13 that the rescan list is not empty, the series of operations proceeds to step S2, and steps S2 to S13 are repeatedly executed until it is determined in step S13 that the rescan list is empty.

Then, in step S13, if it is determined that the rescan list is empty, the network scan operation ends.

Figures 34 and 35 are first and second flowcharts, respectively, for explaining the detailed operation of step S10 in Figure 33.

34, when it is determined in step S8 of Fig. 33 that m = M, the extraction means 14 calculates correlation coefficients r _ij for all communication characteristics of M terminal devices acquired by network scanning by the above-mentioned method (step S100), where i is an argument in the column direction in Figs. 8 to 11, and j is an argument in the row direction in Figs. 8 to 11.

Then, the extraction means 14 sets j=1 (step S101), and sets i=1 (step S102). After that, the extraction means 14 extracts a correlation coefficient r _ij between the communication characteristics f _i and f _j (step S103), and determines whether the extracted correlation coefficient r _ij is greater than a threshold value Corr _r (step S104).

When it is determined in step S104 that the correlation coefficient r _ij is not greater than the threshold value Corr _r , the extracting means 14 deletes the communication characteristics f _i and f _j (step S105).

On the other hand, when it is determined in step S104 that the correlation coefficient r _ij is greater than the threshold value Corr _r , the extracting means 14 stores the communication characteristics f _i and f _j (step S106).

Then, after step S105 or step S106, the extraction means 14 determines whether i=I (I is the maximum number of communication characteristics in the column direction in Figures 8 to 11) (step S107).

If it is determined in step S107 that i is not equal to I, the extraction means 14 sets i to i+1 (step S108). After that, the series of operations proceeds to step S103, and steps S103 to S108 are repeatedly executed until it is determined in step S107 that i is equal to I.

If it is determined in step S107 that i=I, the extraction means 14 determines whether j=J (J is the maximum number of communication characteristics in the row direction in Figures 8 to 11) (step S109).

When it is determined in step S109 that j is not equal to J, the extraction means 14 sets j to j + 1 (step S110). After that, the series of operations proceeds to step S102, and steps S102 to S110 are repeatedly executed until it is determined in step S109 that j is equal to J.

Then, when it is determined in step S109 that j=I, the extraction means 14 calculates a multiple linear regression based on all the stored communication characteristics by the above-mentioned method, with all the stored communication characteristics as explanatory variables and the scan response delay RTT _m as a response variable (step S111), and obtains the contribution rate ^R2 .

Thereafter, the extraction means 14 determines whether the contribution rate ^R2 is smaller than a threshold value _Sr (step S112).

When it is determined in step S112 that the contribution rate ^R2 is smaller than the threshold value _Sr , the extraction means 14 sets the mode of the estimation algorithm ALG to "uncorrelation" (step S113).

On the other hand, when it is determined in step S112 that the contribution rate ^R2 is equal to or greater than the threshold value _Sr , the extraction means 14 sets the mode of the estimation algorithm ALG to "correlation" (step S114).

Then, after step S113 or step S114, the extraction means 14 saves the waiting time and communication characteristics in the network scan and the mode of the estimation algorithm ALG (step S115).

35, after step S115 in FIG. 34, the extraction means 14 calculates a plurality of scores Q of a plurality of communication characteristics having a correlation coefficient r _ij larger than a threshold value Corr _r , and selects K communication characteristics from the plurality of communication characteristics in descending order of score Q (step S116).

Then, the extraction means 14 uses the training data to perform learning of a classification algorithm (e.g., SVM-train) (step S117).

Thereafter, the extraction means 14 sets k=K (step S118). Subsequently, the extraction means 14 classifies the (k×M) sample vectors x ₁₁ to x _kM of the communication characteristics into two groups GP1 and GP2 using the trained classification algorithm (step S119).

Then, the extraction means 14 calculates k weights w ₁ to w _k of the k communication characteristics by using the equations (4), (6), and (7) (step S120).

Thereafter, the extraction means 14 calculates k rank weights R ₁ to R _k based on the k weights w ₁ to w _k using equation (8) (step S121).

Then, the extraction means 14 determines the communication characteristic sample to which the smallest rank weight R _{k_min} is assigned based on the k rank weights R ₁ to R _k as the communication characteristic to be deleted (step S122). After that, the extraction means 14 deletes the communication characteristic sample to which the smallest rank weight R _{k_min} is assigned from the sample vectors x ₁₁ to x _kM (step S123).

Then, the extraction means 14 determines whether the subset of communication characteristics is empty (step S124).

When it is determined in step S124 that the subset of communication characteristics is not empty, the extraction means 14 sets k=k-1 (step S125). That is, the extraction means 14 decreases the number of samples in the subset of communication characteristics by "1". After that, the series of operations proceeds to step S119, and steps S119 to S125 are repeatedly executed until it is determined in step S124 that the subset of communication characteristics is empty.

Then, when it is determined in step S124 that the subset of communication characteristics is empty, the extraction means 14 detects the last deleted communication characteristic as the communication characteristic CMbest _m (step S126).Then, the series of operations proceeds to step S11 in FIG.

After steps S119 to S125 are performed for the first time, steps S119 to S125 are repeatedly performed until the subset of communication characteristics is empty, thereby sequentially deleting samples of communication characteristics that are least important for estimating the results of a network scan. At this time, after deleting one sample of a communication characteristic, the remaining samples are reclassified into two groups GP1 and GP2 using the learned classification algorithm (see step S119). As described above, group GP1 is a group into which communication characteristics that are important for estimating the results of a network scan are classified, and group GP2 is a group into which communication characteristics that are not important for estimating the results of a network scan are classified. Therefore, the accuracy of determining whether or not each sample is a communication characteristic that is important for estimating the results of a network scan is improved, and the sample that is least important for estimating the results of a network scan can be deleted.

The process of detecting the communication characteristic CMbest _m in accordance with the flowcharts shown in FIGS. 34 and 35 constitutes a "detection process."

Moreover, the extraction means 14 that detects the communication characteristic CMbest _m in accordance with the flowcharts shown in FIG. 34 and FIG. 35 constitutes the "detection means."

Figures 36 to 38 are first to third flowcharts, respectively, for explaining the detailed operation of step S11 in Figure 33.

Referring to FIG. 36, after step S10 in FIG. 33, the extraction means 14 detects the standard sample data D_standard_z (z = 1 to Z) used to estimate multiple known estimation results of the network scan from the correspondence table TBL3 stored in the host information database 13 (step S127), and outputs the detected standard sample data D_standard_z (z = 1 to Z) to the calculation means 15.

Then, the extraction means 14 detects sample data x _{i_unknown} acquired in a communication environment where the result of the network scan is unknown from the correspondence table TBL1 (or the correspondence table TBL2) stored in the host information database 13 (step S128). More specifically, the extraction means 14 detects the communication characteristic CMbest _m according to the flowcharts shown in Figures 34 and 35, and detects the scan response delay RTT _m associated with the communication characteristic CMbest _m from the correspondence table TBL1 (or the correspondence table TBL2), thereby detecting the sample data x _{i_unknown} . After that, the extraction means 14 outputs the detected sample data x _{i_unknown} to the calculation means 15.

The calculation means 15 receives the standard sample data D_standard_z (z = 1 to Z) and the sample data x _{i _unknown} from the extraction means 14. Then, the calculation means 15 calculates a probability density function PDF of the scan response delay {RTT _m' _known}_z (m' = 1' to M', z = 1 to Z) of the standard sample data D_standard_z (z = 1 to Z) and the scan response delay RTT _m _unknown (m = 1 to M) of the sample data x _{i _unknown} (step S129).

After that, the calculation means 15 calculates the mean absolute deviation W _unknown of the sample data x _{i _ unknown} (step S130).

Then, the calculation means 15 sets z=1 (step S131), and calculates the mode m(Mo)_known by equation (9) based on the probability density function of the standard sample data D_standard_z (step S132). In this case, when there are two most likely scan response delay values, the calculation means 15 substitutes the two most likely scan response delay values into _r1 and _r2 in equation (9) to calculate the mode m(Mo)_known, and when there are three or more most likely scan response delay values, the calculation means 15 detects the minimum and maximum values from the three or more most likely scan response delays, and substitutes the detected minimum and maximum values into _r1 and _r2 in equation (9) to calculate the mode m(Mo)_known.

Thereafter, the calculation means 15 calculates the relative sample data C(x _{i _known_z} ) by equation (11B) based on the sample data x _{i _known_z} and the mode m(Mo)_known_z (step S133), and calculates the mean absolute deviation W _{known_z} by substituting the calculated relative sample data C(x _{i _known_z} ) into equation (11A) (step S134). That is, the calculation means 15 calculates the mean absolute deviation W _{known_z} by equation (11) based on the sample data x _{i _known_z} and the mode m(Mo)_known_z.

Next, the calculation means 15 calculates the difference S _Sz of the mean absolute deviations based on the mean absolute deviation W _unknown and the mean absolute deviation W _{known_z} by the formula (12) (step S135).

Then, the calculation means 15 calculates the similarity P _Sz based on the difference S _Sz of the mean absolute deviations by using the formula (13) (step S136).

Then, the calculation means 15 judges whether z=Z or not (step S137). When it is judged in step S137 that z=Z is not true, the calculation means 15 sets z=z+1 (step S138). Then, the series of operations proceeds to step S132, and steps S132 to S138 are repeatedly executed until it is judged in step S137 that z=Z.

If it is determined in step S137 that z=Z, the process proceeds to step S139 in FIG. 37.

In the flowchart shown in FIG. 36, when Z=3, steps S133 to S136 are executed as follows:

In step S133 of FIG. 36, when z=1, the calculation means 15 assigns the scan response delay {RTT _m' _known}_1 (m'=1' to M') to x _{i _known_z} , assigns the mode m(Mo)_known_1 to m(Mo)_known_z, and calculates the relative sample data C(x _{i _known_1} ) by the formula (11B); when z=2, the calculation means 15 assigns the scan response delay {RTT _m' _known}_2 (m'=1' to M') to x _{i _known_z} , assigns the mode m(Mo)_known_2 to m(Mo)_known_z, and calculates the relative sample data C(x _{i _known_2 )} by the formula (11B). ) is calculated, and when z=3, the scan response delay {RTT _m' _known}_3 (m'=1' to M') is substituted into x _{i _known_z} , and the mode m(Mo)_known_3 is substituted into m(Mo)_known_z to calculate the relative sample data C(x _{i _known_3} ) according to equation (11B).

Also, in step S134 of FIG. 36 , when z=1, the calculation means 15 substitutes the relative sample data C(x _{i _known —} 1 ) for C(x _{i _known —} z ) to calculate the mean absolute deviation W _{known — 1} according to equation (11A); when z=2, the calculation means 15 substitutes the relative sample data C(x _{i _known — 2} ) for C(x _{i _known — z} ) to calculate the mean absolute deviation W _{known — 2} according to equation (11A); and when z=3, the calculation means 15 substitutes the relative sample data C(x _{i _known — 3} ) for C(x _{i _known — z} ) to calculate the mean absolute deviation W _{known — 3} according to equation (11A).

Furthermore, in step S135 of FIG. 36, when z=1, the calculation means 15 assigns the mean absolute deviation W _{known — 1} to W _{known_z} , assigns the mean absolute deviation W _unknown to W _unknown , and calculates the difference S _S1 of the mean absolute deviations by equation (12), when z=2, assigns the mean absolute deviation W _{known — 2} to W _{known_z} , assigns the mean absolute deviation W _unknown to W _unknown , and calculates the difference S S2 of the mean absolute deviations by equation (12), when z=3, assigns the mean absolute deviation W _{known — 3} to W _{known_z} , assigns the mean absolute deviation W _unknown to W _unknown , and calculates the difference S _S3 of the mean absolute deviations by equation (12). Calculate _S3 .

Furthermore, in step S136 of FIG. 36, when z=1, the calculation means 15 substitutes the difference S _{2 S1} in the mean absolute deviations into S 2 _Sz to calculate the similarity P _{2 S1} by equation (13), when z=2, substitutes the difference S _{2 S2} in the mean absolute deviations into S 2 _Sz to calculate the similarity P 2 _S2 by equation (13), and when z=3, substitutes the difference S _{2 S3} in the mean absolute deviations into S 2 _Sz to calculate the similarity P _{2 S3} by equation (13).

37, when it is determined in step S137 of FIG. 36 that z=Z, the calculation means 15 outputs the similarities P _S1 , P _S2 , . . . , P _SZ to the extraction means 14.

The extraction means 14 receives the similarities P _S1 , P _S2 , ..., P _SZ from the calculation means 15, detects the highest similarity P _{S_highest} among the received similarities P _S1 , P _S2 , ..., P _SZ , and detects the standard sample data D_similarity when the highest similarity P _{S_highest} is obtained (step S139). In this case, when there are multiple highest similarities P _{S_highest} , the extraction means 14 detects an arbitrary similarity P _{S_highest} among the multiple highest similarities P _{S_highest} , and detects the standard sample data D_similarity when the highest similarity P _{S_highest} is obtained.

Then, the extraction means 14 outputs the standard sample data D_similarity when the similarity P _{S_highest} is obtained to the estimation means 16 .

The estimation means 16 receives the standard sample data D_similarity from the extraction means 14. After that, the estimation means 16 sets the F1 score _F1 to "0" (step S140).

Then, the estimation means 16 divides the standard sample data D_similarity into training data and test data (step S141), and divides the training data into a training set and a validation set (step S142).

Then, the estimation means 16 generates initial sample data D_sample_initial for modeling using the above-mentioned method, and trains the estimation algorithm ALG using old training data consisting of the generated initial sample data D_sample_initial (step S143).

Then, the estimation means 16 estimates the results of the network scan by the estimation algorithm ALG (the estimation algorithm ALG after training) using the validation set (step S144), generates a confusion matrix CFM based on the estimated results of the network scan, and outputs it to the calculation means 15.

The calculation means 15 receives the confusion matrix CFM from the estimation means 16, calculates the F1 score _F1 using equations (16) to (18) based on the elements TP, TN, FP, and FN of the received confusion matrix CFM (step S145), and outputs the calculated F1 score _F1 to the estimation means 16.

The estimation means 16 receives the F1 score F ₁ from the calculation means 15. Then, the estimation means 16 divides the training set into subsets d ₁ to d _U (step S146). After that, the series of operations proceeds to step S147 in FIG.

38, after step S146 in FIG. 37, the estimation means 16 sets u=1 (step S147) and trains the estimation algorithm ALG using tentative sample data D_sample_tentative_u including the subset d _u (step S148).

Then, the estimation means 16 estimates the results of the network scan using the validation set (=consisting of all of the subsets V _{1 '} to V _{U '} ) with the estimation algorithm ALG (step S149), and generates a confusion matrix nCFM_u based on the estimated results of the network scan and outputs it to the calculation means 15.

The calculation means 15 receives the confusion matrix nCFM_u from the estimation means 16, calculates the F1 score nF ₁ _u based on the elements TP, TN, FP, and FN of the received confusion matrix nCFM_u using equations (16) to (18) (step S150), and outputs the calculated F1 score nF ₁ _u to the estimation means 16.

The estimation means 16 receives the F1 score nF ₁ _u from the calculation means 15. Then, the estimation means 16 determines whether or not the F1 score nF ₁ _u is greater than the F1 score F ₁ (step S151).

When it is determined in step S151 that the F1 score nF ₁ _u is not greater than the F1 score F ₁ , the series of operations proceeds to step S155.

On the other hand, when it is determined in step S151 that the F1 score nF ₁ _u is greater than the F1 score F ₁ , the estimation means 16 sets the F1 score nF ₁ _u to F ₁ (step S152).

Then, the estimation means 16 sets tentative sample data D_sample_tentative_u including the subset d _u when the F1 score nF ₁ _u is obtained as training data (step S153), and replaces the model variables (variables of the estimation algorithm ALG) with the training data by the above-mentioned method (step S154).

Then, when it is determined in step S151 that the F1 score nF ₁ _u is not greater than the F1 score F ₁ , or after step S154, the estimation means 16 determines whether or not u=U (step S155).

When it is determined in step S155 that u is not equal to U, the estimation means 16 sets u to u+1 (step S156). After that, the series of operations proceeds to step S148, and steps S148 to S156 are repeatedly executed until it is determined in step S155 that u is equal to U.

Then, when it is determined in step S155 that u = U, the estimation means 16 uses the test data (= consisting of all of the subsets TD ₁ to TD ₄ ) to estimate the results of the network scan with a model (estimation algorithm ALG) in which the model variables have been replaced (step S157), and generates a confusion matrix tCFM based on the estimated results of the network scan and outputs it to the calculation means 15.

The calculation means 15 receives the confusion matrix tCFM from the estimation means 16, calculates the F1 score _tF1 using equations (16) to (18) based on the elements TP, TN, FP, and FN of the received confusion matrix tCFM (step S158), and outputs the calculated F1 score _tF1 to the estimation means 16.

The estimation means 16 receives the F1 score _tF1 from the calculation means 15 and determines whether the F1 score _tF1 is equal to or greater than the F1 score _F1 (step S159).

When it is determined in step S159 that the F1 score _tF1 is not equal to _or greater than the F1 score F1, the series of operations proceeds to step S147, and steps S147 to S159 are repeatedly executed until it is determined in step S159 that the F1 score _tF1 is equal to or greater than the F1 score _F1 .

Then, when it is determined in step S159 that the F1 score _tF1 is equal to or greater than the F1 score _F1 , the estimation means 16 approves the model (estimation algorithm ALG) in which the model variables have been replaced and the training data (finally, tentative sample data D_sample_tentative_u including a subset d _u that satisfies _F1 < _{nF1_u} ) (step S160). Note that the tentative sample data D_sample_tentative_u approved in step S160 becomes the data sample D_sample_initial when step S11 in FIG. 33 is executed for the second or subsequent times.

Then, after step S160, the estimation means 16 estimates the result of the network scan by using the sample data x _unknown acquired by the network scan in the communication environment where the result of the network scan is unknown, with the model (estimation algorithm ALG) in which the model variables have been replaced (step S161). After that, the series of operations proceeds to step S12 in FIG. 33.

In steps S140 and S145 in FIG. 37 and S150, S151, S152, S153, S158 and S159 in FIG. 38, Matthews correlation coefficients MCC, nMCC_u and tMCC may be used instead of the F1 scores F ₁ , nF ₁ _u and tF ₁ , respectively.

38 is executed for the first time, in step S159, if it is determined that the F1 score tF ₁ is not equal to or greater than the F1 score F ₁ , and the loop of steps S148 to S156 is executed for the second time, the estimation algorithm ALG in which the model variables have been replaced with optimal training data when the loop of steps S148 to S156 is executed for the first time is used in steps S148 and S149. As a result, when the loop of steps S148 to S156 is executed for the second time, the F1 score nF ₁ _u calculated in step S150 is greater than the F1 score nF ₁ _u calculated in step S150 when the loop of steps S148 to S156 was executed for the first time. Similarly, the F1 score nF ₁ _u calculated in step S150 when the loop from step S148 to step S156 is executed a third, fourth, ... will be greater than the F1 score nF ₁ _u calculated in step S150 when the loop from step S148 to step S156 is executed a second, third, ....

Therefore, as the number of times the loop of steps S148 to S156 is executed increases, the F1 score nF _{1 _u calculated in step S150 in the Rth (R is an integer equal to or greater than 2) execution of the loop of steps S148 to S156 becomes larger than the F1 score nF 1} _u calculated in step S150 in the (R- ₁ )th execution of the loop of steps S148 to S156. Then, in step S154, by replacing the model variables with optimal training data, the estimation algorithm ALG approaches an estimation algorithm when it is estimated that the result of the network scan is a known estimation result (any of "normal", "inadequate communication quality", and "network congestion"). In other words, the estimation algorithm ALG approaches an optimal algorithm.

This increases the likelihood that in step S159, the F1 score _tF1 will be greater than or equal to the F1 score _F1 , and also increases the likelihood that in step S160, the model (estimation algorithm ALG) and training data in which the model variables have been replaced (finally, tentative sample data D_sample_tentative_u including a subset d _u that satisfies F ₁ < nF ₁ _u) will be approved.

Therefore, after it is determined in step S159 that the F1 score _tF1 is not greater than or equal to the F1 score _F1 , the estimation algorithm ALG approaches an optimal algorithm (an algorithm in which the estimation result of the network scan becomes a known estimation result (any of "normal", "poor communication quality", and "network congestion")) without changing the validation set and test data, so that in step S160, the model (estimation algorithm ALG) and training data in which the model variables have been replaced (finally, tentative sample data D_sample_tentative_u including a subset d _u that satisfies _F1 < _{nF1_u} ) are likely to be approved.

Steps S147 to S160 in FIG. 38 correspond to updating the estimation algorithm ALG, which estimates the results of a network scan performed in a communication environment where the results of the network scan are unknown, to approach the optimal algorithm. Here, the optimal algorithm is an algorithm that estimates the results of a network scan such that the estimated results when the results of the network scan are estimated become known estimated results.

After the estimation algorithm ALG approaches the optimal algorithm, in step S161, the result of the network scan is estimated by the estimation algorithm ALG that has approached the optimal algorithm using sample data x _unknown acquired by a network scan in a communication environment where the result of the network scan is unknown, so that the result of the network scan can be accurately estimated.

36, the extraction means 14 extracts standard sample data D_standard_z (z=1 to Z). Then, in steps S131 to S138, the calculation means 15 calculates Z similarities P _S1 to P _SZ between the Z standard sample data D_standard_1 to D_standard_Z and the sample data x _{i _unknown} .

Thereafter, in step S139 of FIG. 37, the extraction means 14 detects the highest similarity P _{S_highest} among the Z similarities P _S1 to P _SZ .

Therefore, by executing steps S127 to S139, the extraction means 14 detects the standard sample data D_similarity (= the standard sample data D_standard_z when the highest similarity P _{S_highest} is obtained) having the probability _density function most similar to the probability density function of the sample data x i _unknown from the Z standard sample data D_standard_1 to standard sample data D_standard_Z.

Thereafter, the estimation means 16 trains the estimation algorithm ALG using old training data (=initial sample data D_sample_initial) (see step S143), and the calculation means 15 uses the validation set to calculate the F1 score _F1 when the results of the network scan are estimated by the trained estimation algorithm ALG (see steps S144 and S145).

Furthermore, the calculation means 15 and the estimation means 16 execute steps S147 to S156 for all of the U pieces of provisional sample data D_sample_tentative_1 to D_sample_tentative_U, each of which includes the U subsets d ₁ to d _U , so that the estimation means 16 acquires the largest F1 score nF ₁ _highest among the U pieces of F1 scores nF ₁ _1 to nF ₁ _U. When it is determined in step S151 that the F1 score nF ₁ _u is greater than _the F1 score F ₁ , the F1 score nF ₁ _u greater than the F1 score F 1 is set as the F1 score F ₁ (see step S152), so that the F1 score F ₁ when step S151 is executed for the uth time is greater than the F1 score F ₁ when step S151 is executed for the (u-1)th time. As a result, the F1 score nF ₁ _u determined to be greater than the F1 score F ₁ by the u-th execution of step S151 is greater than the F1 score nF 1 _u determined to be greater than the F1 score F ₁ by the (u- ₁ )-th execution of step S151. Therefore, by executing steps S147 to S156 for all of U pieces of provisional sample data D_sample_tentative_1 to D_sample_tentative_U each including U subsets d ₁ to d _U , the largest F1 score nF ₁ _highest among the U F1 scores nF ₁ _1 to nF ₁ _U is obtained.

Then, the estimation means 16 sets the provisional sample data D_sample_highest including the subset d _{u _highest} when the highest F1 score nF ₁ _highest is obtained among the U F1 scores nF ₁ _1 to nF ₁ _U as the training data (see step S153), and replaces the model variables (variables of the estimation algorithm ALG) with the set training data (see step S154).

As a result, the F1 score nF ₁ _highest indicates the highest accuracy in estimating the results of the network scan, and the variables of the estimation algorithm ALG are replaced with the hypothetical sample data D_sample_highest which includes the subset d _{u _highest} when the F1 score nF ₁ _highest is obtained, so that the accuracy in estimating the results of the network scan can be maximized by estimating the results of the network scan using the estimation algorithm ALG having variables replaced with the hypothetical sample data D_sample_highest.

In addition, since the standard sample data D_standard_z (z = 1 to Z) consists of a scan response delay {RTT _m' _known}_z (m' = 1' to M', z = 1 to Z) and a communication characteristic {CMbest _m' _known}_z (m' = 1' to M', z = 1 to Z), the process in which the extraction means 14 detects the standard sample data D_similarity (= the standard sample data D_standard_z when the highest similarity P _{S_highest} is calculated) having a probability density function most similar to the probability density function of the sample data x _{i _unknown} from the Z pieces of standard sample data D_standard_1 to standard sample data D_standard_Z constitutes a "detection process". Furthermore, the scan response delay {RTT _m' _known}_similarity (m' = 1' to M') included in the standard sample data D_similarity constitutes a "similar scan response delay." Furthermore, the extraction means 14 that executes the detection process constitutes a "detection means." Furthermore, since each of M' and Z is an integer equal to or greater than 2, {RTT _m' _known}_z (m' = 1' to M', z = 1 to Z) included in the standard sample data D_standard_z (z = 1 to Z) constitutes "multiple sets of M' second scan response delays."

Moreover, since the sample data x _{i _unknown} is composed of a scan response delay RTT _m _unknown (m = 1 to M) and a communication characteristic CMbest _m _unknown (m = 1 to M), it constitutes “first sample data". Then, the extraction means 14 that detects the communication characteristic CMbest _m according to the flowcharts shown in Figures 34 and 35 and detects the scan response delay RTT _m combined with the communication characteristic CMbest _m from the correspondence table TBL1 (or correspondence table TBL2) to detect the sample data x _{i _unknown} constitutes “generation means".

Furthermore, the standard sample data D_similarity is made up of scan response delay {RTT _m '_known}_similarity (m'=1' to M') and communication characteristic {CMbest _{m'_known} }_similarity (m'=1' to M'), and therefore constitutes "similar standard sample data". The extraction means 14 that detects the standard sample data D_similarity constitutes the "generation means".

Furthermore, the F1 scores F ₁ , nF ₁ (=nF ₁ _1 to nF ₁ _U), tF ₁ or the Matthews correlation coefficients MCC, nMCC (=nMCC_1 to nMCC_U), and tMCC constitute an "evaluation index", the F1 score F ₁ or the Matthews correlation coefficient MCC constitutes a "first evaluation index", the F1 scores nF ₁ (=nF ₁ _1 to nF ₁ _U) or the Matthews correlation coefficient nMCC (=nMCC_1 to nMCC_U) constitute a "second evaluation index", and the F1 score tF ₁ or the Matthews correlation coefficient tMCC constitutes a "third evaluation index".

Furthermore, the estimation means 16 that replaces the variables of the estimation algorithm ALG with the training data (=the tentative sample data D_sample_tentative including the subset d _{u_highest} ) constitutes an "update means" that updates the estimation algorithm ALG with the first sample data.

Furthermore, steps S132 to S136 are performed for all z (=1 to Z), and the standard sample data D_similarity when the highest similarity P _{S_highest} is obtained in step S139 is detected by detecting M' scan response delays {RTT _m' _unknown}_1 to {RTT m' _unknown}_Z having a distribution most similar to the distribution of M scan response delays RTT _m _unknown based on the distribution of Z scan response delays {RTT _m' _known}_z (m'=1' to M', z=1 to Z) of the standard sample data _{D_standard_z} ( _z =1 to Z) and the distribution of the scan response delay RTT _m _unknown (m=1 to M) of the sample data x i _unknown. This corresponds to detecting {RTT _{1 '} _known}_z through {RTT _{M '} _known}_z as similar scan response delays from a plurality of sets (=Z sets) of M' scan response delays {RTT _{1 '} _known}_z through {RTT _{M '} _known}_z.

Furthermore, calculating the mean absolute deviation W _unknown in step S130 constitutes a "first calculation process."

Furthermore, calculating the similarity P _Sz in steps S132 to S136 constitutes a "second calculation process."

Furthermore, executing step S148 constitutes executing a "first training", and executing step S143 constitutes executing a "second training". And the estimation means 16 that executes steps S143 and S148 constitutes a "training means".

Figure 39 is a flowchart explaining the detailed operation of step S130 in Figure 36.

39, after step S129 in FIG. 36, the calculation means 15 calculates the mode m(Mo)_unknown of the sample data x _{i_unknown} by equation (9) based on the probability density function of the sample data x _{i_unknown} (step S1301).

Thereafter, the calculation means 15 calculates the relative sample data C( _{xi_unknown} )_unknown by the formula (10B) based on the sample data _{xi_unknown} and the mode m(Mo)_unknown (step S1302), and substitutes the calculated relative sample data C( _{xi_unknown} )_unknown into the formula (10A) to calculate the mean absolute deviation _Wunknown (step S1303). That is, the calculation means 15 calculates the mean absolute deviation _Wunknown by the formula (10) based on the sample data _{xi_unknown} and the mode m(Mo)_unknown. Then, after step S1303, the series of operations proceeds to step S131 in FIG. 36.

Figures 40 to 42 are first to third flowcharts, respectively, for explaining the detailed operation of step S143 in Figure 37.

40, after step S142 in FIG. 37, the estimation means 16 detects the scan response delay {RTT _m' _known}_similarity and the communication characteristic {CMbest _m' _known}_similarity included in the standard sample data D_similarity received from the extraction means 14.

Then, the estimation means 16 sets b=1 (step S1431) and generates a discriminant function δ _g (x) using the LDA algorithm (step S1432). Subsequently, the estimation means 16 sets a model Model for estimating the result of the network scan to Model=1 (step S1433), sets m _b =1 (step S1434), and sets DS_count=0 (step S1435). Here, m _b represents a terminal device belonging to one base station b.

Then, the estimation means 16 calculates the D-score of the terminal device having the communication characteristic {CMbest _m' _known}_similarity by using the discriminant function δ _g (x) (step S1436).

Then, the estimation means 16 determines whether the D score is less than zero (0) (step S1437).

When it is determined in step S1437 that the D score is less than zero (0), the estimation means 16 sets DS_count = DS_count + 1 (step S1438). That is, the estimation means 16 increments the count value DS_count by "1".

Then, when it is determined in step S1437 that the D score is not smaller than zero (0), or after step S1438, the estimation means 16 determines whether or not m _b =N _terminal (step S1439).

When it is determined in step S1439 that m _b is not equal to N _terminal , the estimation means 16 sets m _b =m _b +1 (step S1440). After that, the series of operations proceeds to step S1436, and steps S1436 to S1440 are repeatedly executed until it is determined in step S1439 that m _b =N _terminal .

Then, when it is determined in step S1439 that m _b =N _terminal , the estimation means 16 determines whether or not Model = 1 (step S1441).

When it is determined in step S1441 that Model=1, the estimation means 16 determines whether or not DS_count=0, or whether or not DS_count>(N _terminal ×D_thres) (step S1442).

When it is determined in step S1442 that DS_count is not 0 and DS_count is not greater than (N _terminal ×D_thres), the estimation means 16 sets Model=3 (step S1443).

On the other hand, when it is determined in step S1442 that DS_count=0 or DS_count>(N _terminal ×D_thres), the estimation means 16 sets Model=2 (step S1444).

Then, after step S1443 or step S1444, the series of operations proceeds to step S1434.

On the other hand, if it is determined in step S1441 that Model is not 1, the process proceeds to step S1445 in FIG. 41.

Referring to FIG. 41, when it is determined in step S1441 that Model is not 1, the estimation means 16 determines whether DS_count is 0 or not (step S1445).

When it is determined in step S1445 that DS_count = 0, the estimation means 16 determines that the scan data for the terminal device is "normal" (step S1446).

On the other hand, when it is determined in step S1445 that DS_count is not 0, the estimation means 16 determines whether or not DS_count<(N _terminal ×D_thres) (step S1447).

When it is determined in step S1447 that DS_count<(N _terminal ×D_thres) is not satisfied, the estimation means 16 estimates that the cause of the failure of the network scan for all terminal devices belonging to base station b is "network congestion" (step S1448).

On the other hand, when it is determined in step S1447 that DS_count < (N _terminal × D_thres), the estimation means 16 estimates that the cause of the failure of the network scan for a terminal device having a D score smaller than "0" is "insufficient SNR (insufficient communication quality)" (step S1449).

Then, after any of steps S1446, S1448, and S1449, the series of operations proceeds to step S1450 in FIG. 42.

Referring to FIG. 42, after any of step S1446, step S1448, and step S1449 in FIG. 41, the estimation means 16 sets m _b =1 (step S1450).

Then, the estimation unit 16 judges whether or not the result of the network scan for the terminal device m- _b has been estimated (step S1451).

When it is determined in step S1451 that the result of the network scan for the terminal device _mb has not been estimated, the estimation means 16 determines whether or not there is no scan response for all ports of the terminal device _mb (step S1452).

When it is determined in step S1452 that there is a scan response from some of the ports of the terminal device _mb , the estimation means 16 determines whether or not there is a scan response from all of the ports of the terminal device _mb (step S1453).

When it is determined in step S1453 that there is a scan response from all ports of the terminal device _mb , the estimation means 16 estimates that the network scan for the terminal device _mb is "normal" (step S1454).

On the other hand, when it is determined in step S1453 that there is no scan response from some ports of the terminal device _mb , the estimation means 16 estimates that the cause of the failure of the network scan for the terminal device _mb is "port closed" (step S1455).The estimation means 16 then stores the obtained waiting time in the host information database 13 (step S1456).

On the other hand, when it is determined in step S1452 that there is no scan response from all ports of the terminal device _mb , the estimation means 16 estimates that the cause of the failure of the network scan for the terminal device _mb is "no network connection" (step S1457).

Then, when it is determined in step S1451 that the result for the terminal device m _b has been estimated, or after any of steps S1454, S1456 and S1457, the estimation means 16 determines whether or not m _b =N _terminal (step S1458).

When it is determined in step S1458 that m _b is not equal to N _terminal , the estimation means 16 sets m _b =m _b +1 (step S1459). After that, the series of operations proceeds to step S1451, and steps S1451 to S1459 are repeatedly executed until it is determined in step S1458 that m _b =N _terminal .

Then, when it is determined in step S1458 that m _b =N _terminal , the estimation means 16 determines whether or not b =B (step S1460).

When it is determined in step S1460 that b is not equal to B, the estimation means 16 sets b to b+1 (step S1461). After that, the series of operations proceeds to step S1432 in FIG. 40, and steps S1432 to S1461 are repeatedly executed until it is determined in step S1460 that b is equal to B.

If it is determined in step S1460 that b=B, the process proceeds to step S144 in FIG. 37.

The detailed operations of step S144 in FIG. 37 and steps S148, S149, S157, and S161 in FIG. 38 are also performed according to the flowcharts shown in FIG. 40 to FIG. 42.

When the flowcharts shown in Figures 40 to 42 are executed as detailed operations of step S144 in Figure 37, if the estimation means 16 determines in step S1460 that b = B, it generates a confusion matrix CFM and outputs the generated confusion matrix CFM to the calculation means 15.

When the flowcharts shown in Figures 40 to 42 are executed as detailed operations of step S149 in Figure 38, if the estimation means 16 determines in step S1460 that b = B, it generates a confusion matrix nCFM_u and outputs the generated confusion matrix nCFM_u to the calculation means 15.

Furthermore, when the flowcharts shown in Figures 40 to 42 are executed as detailed operations of step S157 in Figure 38, if the estimation means 16 determines in step S1460 that b = B, it generates a confusion matrix tCFM and outputs the generated confusion matrix tCFM to the calculation means 15.

Steps S1450 to S1459 in FIG. 42 are steps for estimating the results of a network scan (either "normal," "port closed," or "no network connection") for each of all terminal devices belonging to one base station.

Fig. 43 is a flowchart for explaining the detailed operation of step S1432 in Fig. 40. Note that Fig. 43 explains that the discriminant function δ _g (x) is generated by a trained LDA algorithm trained using scan data when the network scan is "normal" (communication characteristics CMbest when the "network is congested").

43, after step S1431 in FIG. 40 or step S1461 in FIG. 42, the estimation means 16 classifies terminal devices having communication characteristics CMbest into G (G=3) groups CLS ₁ to CLS ₃ using the LDA algorithm (step S1432-1).

Then, the estimation means 16 calculates the averages μ ₁ , μ ₂ , and μ ₃ of each group using the formula (20C) (or the formula (20D)) (step S1432-2).

Thereafter, the estimation means 16 calculates the prior averages Π ₁ , Π ₂ , Π ₃ of each group using equation (20E) (or equation (20F)) (step S1432-3).

The estimation means 16 then calculates the covariance matrix C of each group using equation (20B) (step S1432-4).

Then, the estimation means 16 calculates the joint covariance matrix _C12 of Model1 by equation (21) (step S1432-5). The estimation means 16 also calculates the joint covariance matrix _C13 of Model2 by equation (22) (step S1432-6). Furthermore, the estimation means 16 calculates the joint covariance matrix _C23 of Model3 by equation (23) (step S1432-7).

The estimation means 16 then generates a discriminant function δ _pq (x) using equation (20A) (step S1432-8). The joint covariance matrices C ₁₂ , C ₁₃ , and C ₂₃ calculated in steps S1432-5 to S1432-7 are substituted for “C” in equation (20A).

In the flow chart shown in FIG. 43, the estimation means 16 classifies, for example, terminal devices having communication characteristics CMbest into three groups CLS ₁ to CLS ₃ in step S1432-1.

As described above, group CLS ₁ is a "normal" group, group CLS ₂ is a "poor communication quality" group, and group CLS ₃ is a "network congestion" group. Model 1 handles groups CLS ₁ and CLS ₂ , Model 2 handles groups CLS ₁ and CLS ₃ , and Model 3 handles groups CLS ₂ and CLS ₃ .

Therefore, when Model1 is used, the joint covariance matrix _C12 calculated in step S1432-5 is substituted for "C" in step S1432-8. When Model2 is used, the joint covariance matrix _C13 calculated in step S1432-6 is substituted for "C" in step S1432-8. Furthermore, when Model3 is used, the joint covariance matrix _C23 calculated in step S1432-7 is substituted for "C" in step S1432-8.

Furthermore, since the failure causes of "inadequate communication quality" and "network congestion" are failure causes caused by the communication channel with the terminal device, as described above, classifying the terminal devices having the communication characteristic CMbest into three groups CLS ₁ to CLS ₃ in step S1432-1 corresponds to "classifying into a first group indicating that the network scan is normal, and into second and third groups indicating that the network scan has failed due to a cause caused by the communication channel with the terminal device." In this case, group CLS ₂ constitutes the second group, and group CLS ₃ constitutes the third group.

Figure 44 is a flowchart explaining the detailed operation of step S1436 in Figure 40.

Referring to FIG. 44, after step S1435 or step S1440 in FIG. 40, the estimation means 16 determines whether Model=1 (step S1436-1).

When it is determined in step S1436-1 that Model=1, the estimation means 16 substitutes the joint covariance matrix _C12 calculated in step S1432-5 of Fig. 43 for _C12 in formula (24A) to calculate a discriminant function ^f1 (step S1436-2). This discriminant function ^f1 is a discriminant function expressed using the joint covariance matrix _C12 obtained by combining the covariance matrix of group _CLS1 (= _g1 ) and the covariance matrix of group _CLS2 (= _g2 ), the mean _μ1 of group _CLS1 (= _g1 ), _and the prior mean _Π1 of group CLS1 (= _g1 ).

Thereafter, the estimation means 16 substitutes the joint covariance matrix _C12 calculated in step S1432-5 of Fig. 43 for _C12 in formula (24B) to calculate a discriminant function ^f2 (step S1436-3). This discriminant function ^f2 is a discriminant function expressed using the joint covariance matrix _C12 obtained by combining the covariance matrix of the group CLS1 (= _g1 ) and the covariance matrix of the group _CLS2 (= _g2 ), _the mean _μ2 of the group _CLS2 (= _g2 ), and the prior mean _Π2 of the group _CLS2 (= _g2 ).

On the other hand, if it is determined in step S1436-1 that Model is not 1, the estimation means 16 determines whether Model is 2 (step S1436-4).

When it is determined in step S1436-4 that Model=2, the estimation means 16 substitutes the joint covariance matrix _C13 calculated in step S1432-6 of Fig. 43 for _C13 in formula (25A) to calculate a discriminant function ^f1 (step S1436-5). This discriminant function ^f1 is a discriminant function expressed using the joint covariance matrix _C13 obtained by combining the covariance matrix of group _CLS1 (= _g1 ) and the covariance matrix of group _CLS3 (= _g3 ), the mean _μ1 of group _CLS1 (= _g1 ), _and the prior mean _Π1 of group CLS1 (= _g1 ).

Thereafter, the estimation means 16 substitutes the joint covariance matrix _C13 calculated in step S1432-6 of Fig. 43 for _C13 in formula (25B) to calculate a discriminant function ^f2 (step S1436-6). This discriminant function ^f2 is a discriminant function expressed using the joint covariance matrix _C13 obtained by combining the covariance matrix of the group _CLS1 (= _g1 ) and the covariance matrix of the group _CLS3 (= _g3 ), the mean _μ3 of the group _CLS3 (= _g3 ), and the prior mean _Π3 of the group _CLS3 (= _g3 ).

On the other hand, when it is determined in step S1436-4 that Model=2 is not satisfied, the estimation stage 15 substitutes the joint covariance matrix C23 calculated in step S1432-7 of Fig. ₄₃ for _C23 in equation (26A) to calculate a discriminant function ^f1 (step S1436-7). This discriminant function ^f1 is a discriminant function expressed using the joint covariance matrix _C23 obtained by combining the covariance matrix of group _CLS2 (= _{g2 )} and _the covariance matrix of group _CLS3 (= _g3 ), the mean _μ2 of group CLS2 (= _g2 ), and the prior mean _Π2 of group _CLS2 (=g2).

Thereafter, the estimation means 16 substitutes the joint covariance matrix _C23 calculated in step S1432-7 of Fig. 43 for _C23 in formula (26B) to calculate a discriminant function ^f2 (step S1436-8). This discriminant function ^f2 is a discriminant function expressed using the joint covariance matrix C23 obtained by combining the covariance matrix of group _CLS2 (= _{g2 )} and the covariance matrix of group _CLS3 (= _{g3 )} , the mean _μ3 of group _CLS3 (= _g3 ), and the prior mean _Π3 of group CLS3 (= _g3 ).

Then, after any of step S1436-3, step S1436-6, and step S1436-8, the estimation means 16 calculates D score=f ¹ −f ² (step S1436-9). After that, the series of operations proceeds to step S1437 in FIG.

In the flowchart shown in FIG. 44, Model=1 is set in step S1433 in FIG. 40, so when step S1436 in FIG. 40 is executed first, it is determined in step S1436-1 in FIG. 44 that Model=1, and after steps S1436-2 and S1436-3 are executed in sequence, the D score is calculated in step S1436-9 using Model1.

Then, after step S1436-9, steps S1436 to S1440 in FIG. 40 are repeatedly executed until it is determined in step S1439 in FIG. 40 that m _b =N _terminal , and then in step S1441 it is determined that Model = 1, and steps S1442 and S1443 are executed in sequence, or steps S1442 and S1444 are executed in sequence.

When steps S1442 and S1443 are executed in sequence, Model = 3 is set (see step S1443), so when step S1436 is executed after moving from step S1443 to step S1434, it is determined in step S1436-1 of FIG. 44 that Model = 1 is not true, and further, in step S1436-4, it is determined that Model = 2 is not true, and after steps S1436-7 and S1436-8 are executed in sequence, in step S1436-9, the D score is calculated using Model 3.

On the other hand, if steps S1442 and S1444 in FIG. 40 are executed in sequence, Model=2 is set (see step S1444), and when step S1436 is executed after moving from step S1444 to step S1434, it is determined in step S1436-1 in FIG. 44 that Model=1 is not true, and further, in step S1436-4, it is determined that Model=2 is true, and after steps S1436-5 and S1436-6 are executed in sequence, in step S1436-9, the D score is calculated using Model2.

Therefore, the flowchart shown in Figure 44 is a flowchart for calculating the D score according to the set Model (either Model 1, Model 2, or Model 3).

In addition, in the flowcharts shown in Figures 40 to 42 for estimating the results of a network scan, after setting Model = 1 in step S1433, the D score is calculated in step S1436 because, by using Model 1, it is first possible to estimate whether the network scan is "normal" or "failed (= failed due to insufficient communication quality)", and when it is estimated that the network scan is "failed (= failed due to insufficient communication quality)", the cause of the network scan failure can be further estimated.

In addition, in step S1433, after Model=1 is set, D scores are calculated for all terminal devices belonging to one base station b using Model1, and the number of D scores smaller than "0" is counted as the count value DS_count (see steps S1436 to S1440).

Then, when step S1441 is executed for the first time, Model=1 is set, so in step S1441 it is determined that Model=1, and in steps S1442 to S1444, Model=2 or Model=3 is set.

Therefore, Model2 or Model3 is set according to the number of D scores (=DS_count) calculated using Model1. In other words, Model2 or Model3 is set based on Model1.

Then, in steps S1442 to S1444, after Model=2 or Model=3 is set, the D-score is calculated using Model2 or Model3, and the number of D-scores smaller than "0" is counted as a count value DS_count (see steps S1436 to S1440). More specifically, when Model=2 is set, the D-score is calculated using the communication characteristics CMbest classified into group _CLS1 , the communication characteristics CMbest classified into group _CLS3 , and the joint covariance matrix _C13 shown in formula (22), and when Model=3 is set, the D-score is calculated using the communication characteristics CMbest classified into group _CLS2 , the communication characteristics CMbest classified into group _CLS3 , and the joint covariance matrix C23 shown in formula ( ₂₃ ). Then, based on the calculated D scores, the number of D scores less than "0" is counted as a count value DS_count.

Then, in step S1441, it is determined that Model=1 is not satisfied, and in steps S1445 to S1457 shown in Figures 41 and 42, Model 2 or Model 3 is used to estimate the result of the network scan (either "normal," "poor communication quality," "network congestion," "port closed," or "no network connection").

When Model2 is set and the results of a network scan are estimated, the number of D scores calculated using Model2 that are smaller than "0" (= count value DS_count) is used.

In addition, when Model 3 is set and the results of a network scan are estimated, the number of D scores calculated using Model 3 that are smaller than "0" (= count value DS_count) is used.

In this embodiment of the invention, the estimation algorithm ALG is composed of steps S1431 to S1461 shown in Figures 40 to 42 (including the flowcharts shown in Figures 43 and 44).

Figure 45 is a flowchart explaining the detailed operation of step S145 in Figure 37.

Referring to FIG. 45, after step S144 in FIG. 37, the calculation means 15 receives OE, ECE, and OIE from the estimation means 16, and detects the number of OE, ECE, and OIE received (step S145-1).

Then, the calculation means 15 calculates TP using formula (14A) based on the number of OEs and the number of ECEs (step S145-2).

Then, the calculation means 15 calculates the FN using formula (14B) based on the number of OIEs and the number of ECEs (step S145-3).

The calculation means 15 then calculates FP using TP according to equation (15A) (step S145-4).

Then, the calculation means 15 calculates Precision using formula (16) based on the proportion classified as TP and the proportion classified as FP (step S145-5). The calculation means 15 also calculates Recall using formula (17) based on the proportion classified as TP and the proportion classified as FN (step S145-6).

Thereafter, the calculation means 15 calculates the F1 score _F1 by substituting Precision and Recall into the formula (18) (step S145-7). Then, the series of operations proceeds to step S146 in FIG.

Note that the detailed operations of steps S150 and S158 in Fig. 38 are also executed according to the flowchart shown in Fig. 45. When the detailed operation of step S150 is executed, calculation means 15 receives OE, ECE and OIE from estimation means 16, receives the received OE, ECE and OIE from estimation means 16, receives the received OE, ECE and OIE from estimation means 16, and sequentially executes steps S145-1 to S145-7 based on the received OE, ECE and OIE to calculate F1 score nF ₁ _u. Furthermore, when the detailed operation of step S158 is executed, the calculation means 15 receives the OE, ECE, and OIE from the estimation means 16, receives the received OE, ECE, and OIE from the estimation means 16, receives the received OE, ECE, and OIE from the estimation means 16, and sequentially executes steps S145-1 to S145-7 based on the received OE, ECE, and OIE to calculate the F1 score _tF1 .

Moreover, calculating the F1 scores F ₁ , nF ₁ _u, tF ₁ in accordance with steps S145-1 to S145-7 constitutes a "third calculation process."

Figure 46 is a flowchart for explaining the detailed operation of step S145 when the Matthews correlation coefficient MCC is calculated in step S145 of Figure 37.

The flowchart shown in FIG. 46 is the same as the flowchart shown in FIG. 45, except that steps S144-4 to S145-7 in the flowchart shown in FIG. 45 have been replaced with steps S145-4A and S145-5A.

Referring to FIG. 46, when the detailed operation of step S145 in FIG. 37 is started, the calculation means 15 sequentially executes steps S145-1 to S145-3 described above. Then, after step S145-3, the calculation means 15 calculates FP using TP according to equation (15A), and calculates TN using FN according to equation (15B) (step S145-4A).

Then, the calculation means 15 substitutes TP, TN, FP, and FN into equation (19) to calculate the Matthews correlation coefficient MCC (step S145-5A). After that, the series of operations proceeds to step S146 in FIG. 37.

When the Matthews correlation coefficient MCC is calculated in steps S150 and S158 of FIG. 38, the detailed operations of steps S150 and S158 are also executed according to the flowchart shown in FIG. 46. When the detailed operation of step S150 is executed, the calculation means 15 sequentially executes steps S145-1 to S145-3, S145-4A, and S145-5A to calculate the Matthews correlation coefficient nMCC_u. When the detailed operation of step S158 is executed, the calculation means 15 sequentially executes steps S145-1 to S145-3, S145-4A, and S145-5A to calculate the Matthews correlation coefficient tMCC.

Fig. 47 shows examples of confusion matrices, where (a) in Fig. 47 shows the confusion matrix when the results of a network scan are estimated by the estimation algorithm ALG using a validation set (=consisting of all of the subsets V _{1 '} -V _{8 '} ), and (b) in Fig. 47 shows the confusion matrix when the results of a network scan are estimated by the estimation algorithm ALG using test data (=consisting of all of the subsets TD ₁ -TD ₄ ).

Referring to (a) of FIG. 47, when the result of the network scan is “normal” (known estimated result), when the result of the network scan is estimated by the estimation algorithm ALG using the validation set (=consisting of all of subsets V1 _′ to V8 _′ ), a confusion matrix consisting of, for example, TP=64, FN=0, TN=0 and FP=0 is obtained.

Referring to FIG. 47(b), when the result of the network scan is “normal” (known estimation result), when the result of the network scan is estimated by the estimation algorithm ALG using the test data (=consisting of all of the subsets TD ₁ to TD ₄ ), a confusion matrix consisting of, for example, TP=32, FN=0, TN=0, and FP=0 is obtained.

Therefore, in step S145 of FIG. 37 and step S150 of FIG. 38, the confusion matrix shown in FIG. 47(a) (the confusion matrix when the result of the network scan is estimated by the estimation algorithm ALG using the validation set (=consisting of all of the subsets V1 _' to V8 _' )) is generated, and in step S158 of FIG. 38, the confusion matrix shown in FIG. 47(b) (the confusion matrix when the result of the network scan is estimated by the estimation algorithm ALG using the test data (=consisting of all of the subsets _TD1 to _TD4 )) is generated.

Figure 48 shows the results of a simulation of the D score. As a result of performing a simulation of the D score with reference to Figure 48, it was found that when the network scan was "normal", the D score was greater than "0" for more than half of the terminal devices (see (a) of Figure 48).

In addition, when the cause of a network scan failure was "insufficient communication quality," it was found that the D score was less than "0" in less than half of the terminal devices (see (b) in Figure 48).

Furthermore, when the cause of the network scan failure was "network congestion," it was found that the D score was less than "0" in more than half of the terminal devices (see (c) in Figure 48).

Therefore, by using the number of D scores less than "0" (= count value DS_count), it is possible to estimate whether the results of the network scan are "normal," "inadequate communication quality," or "network congestion."

Model 2 handles group CLS ₁ where the network scan is “normal” and group CLS ₃ where the network scan failed due to “network congestion”, and calculates a D-score D_2 to distinguish between group CLS ₁ and group CLS _3. If the number of D-scores D_2 that are smaller than “0” (=count value DS_count) among the calculated D-scores D_2 is “0”, it can be presumed that the network scan is “normal”, and if the number of D-scores D_2 that are smaller than “0” (=count value DS_count) is greater than half of the terminal devices, it can be presumed that the network scan failed due to “network congestion”.

Model 3 handles group CLS ₂ in which the network scan has failed due to “inadequate communication quality” and group CLS _{3 in which the network scan has failed due to “network congestion”, and calculates a D-score D_3 to distinguish between group CLS 2 and} group CLS _3. If the number of D-scores D_3 less than “0” (=count value DS_count) among the calculated D-scores D_3 is less than half of the terminal devices, it can be assumed that the network scan has failed due to “inadequate communication quality”, and if the number of D-scores D_3 less than “0” (=count value DS_count) is greater than half of the terminal devices, it can be assumed that the network scan has failed due to “network congestion”.

Therefore, first, in step S1445 of FIG. 41, it is determined whether DS_count=0 or not. If it is determined that DS_count=0 is not true, it is further determined whether DS_count<(N _terminal ×D_thres) or not. If DS_count<(N _terminal ×D_thres), it is presumed that the network scan failed for terminal devices having a D score smaller than "0" due to "insufficient communication quality". If DS_count<(N _terminal ×D_thres) is not true (i.e., when DS_count is equal to or greater than (N _terminal ×D_thres)), it is presumed that the network scan failed for all terminal devices belonging to the base station due to "network congestion" (see steps S1445, S1447 to S1449).

When Model 2 is set, in steps S1445 and S1447 to S1449, the D-score D_2 is used to distinguish between group CLS ₁ in which the network scan is “normal” and group CLS ₃ in which the network scan has failed due to “network congestion”, so that, depending on the number of D-scores D_2 that are smaller than “0” (=count value DS_count), it is possible to accurately estimate whether DS_count=0 (i.e., the network scan is “normal”) or DS_count≠0 (i.e., the network scan has failed due to “inadequate communication quality” or “network congestion”). Then, when DS_count is not 0 (i.e., DS_count ≠ 0), depending on whether the count value DS_count, which is the number of D scores D_2 smaller than "0", is smaller than (N _terminal × D_thres), it can be accurately estimated that the cause of the failure of the network scan is "insufficient communication quality" or "network congestion".

On the other hand, when Model 3 is set, in steps S1445, S1447 to S1449, a D-score D_3 is used to distinguish between group CLS ₂ in which the network scan has failed due to "inadequate communication quality" and group CLS ₃ in which the network scan has failed due to "network congestion". Therefore, when DS_count is not 0 (i.e., DS_count ≠ 0), it is possible to accurately estimate that the cause of the network scan failure is "inadequate communication quality" or "network congestion" depending on whether the count value DS_count, which is the number of D-scores D_3 smaller than "0", is smaller than (N _terminal ×D_thres). In addition, since the determination of whether DS_count=0 or not is clear, it is possible to accurately estimate whether DS_count=0 (i.e., the network scan is “normal”) or whether DS_count≠0 (i.e., the network scan has failed due to “inadequate communication quality” or “network congestion”).

Therefore, whether Model 2 or Model 3 is used, it is possible to accurately estimate whether a network scan is "normal" or "failed." When it is estimated that a network scan is "failed," it is also possible to accurately estimate that the cause of the network scan failure is "insufficient communication quality" or "network congestion."

Furthermore, when it is determined in step S1445 that DS_count = 0, the network scan is not immediately assumed to be "normal", but is instead assumed to be either "normal", a failure due to a "closed port", or a failure due to "no network connection" for each terminal device port, depending on the number of ports for which a scan response was received (see steps S1452 to S1455 and S1457). Therefore, by detecting the presence or absence of a scan response at each port, it is possible to accurately estimate whether the network scan was "normal", a failure due to a "closed port", or a failure due to "no network connection".

Furthermore, when the process proceeds from "YES" in step S1447 to step S1449, the results of the network scan have not been estimated for terminal devices having D scores not smaller than "0" (i.e., D scores greater than or equal to "0" and smaller than (N _terminal ×D_thres)) in step S1449. However, after step S1449, steps S1452 to S1457 in FIG. 42 are executed for each terminal device m- _b for which the results of the network scan have not been estimated. Therefore, the results of the network scan for terminal devices having D scores not smaller than "0" (i.e., D scores greater than or equal to "0" and smaller than (N _terminal ×D_thres)) are estimated to be "normal", the cause of the network scan failure is "port closed", or the cause of the network scan failure is "no network connection" (see steps S1454, S1455, S1457).

The steps consisting of "YES" in step S1445 → "NO" in S1451 → "NO" in S1452 → "YES" in S1453 → S1454, and "NO" in step S1445 → S1447 to S1449 correspond to estimating that the network scan is successful or unsuccessful based on the D score D_2 or the D score D_3, and when it is estimated that the network scan has failed, estimating that the network scan has failed due to a cause arising from the communication channel with the terminal device.

In an embodiment of the present invention, the operation of estimating the results of a network scan may be realized by software. In this case, the network scanning device 1 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The ROM stores a program Prog_A consisting of the steps of the flowchart shown in FIG. 33 (including the flowcharts shown in FIG. 34 to FIG. 44 and the flowchart shown in FIG. 45 or the flowchart shown in FIG. 46).

The CPU reads the program Prog_A from the ROM and executes the read program Prog_A to perform an operation to estimate the results of the network scan. The RAM temporarily stores the various calculation results described above.

The program Prog_A may also be distributed in a form recorded on a recording medium such as a CD or DVD. When the recording medium on which the program Prog_A is recorded is inserted into a computer, the computer reads and executes the program Prog_A from the recording medium, and performs an operation to estimate the results of a network scan.

Therefore, the recording medium on which the program Prog_A is recorded is a computer-readable recording medium.

In the above, it has been described that when classifying terminal devices having communication characteristics CMbest into groups CLS ₁ to CLS ₃ , the estimation means 16 learns an LDA algorithm using scan data when the network scan is “network congested” (communication characteristics CMbest when “network is congested”), and classifies terminal devices having communication characteristics CMbest into three groups CLS ₁ to CLS ₃ using the learned LDA algorithm. However, in the embodiment of the present invention, it is not limited to this, and the estimation means 16 may learn an LDA algorithm using scan data when the most accurately estimated result is obtained from the results of the network scan (= either “normal”, “network congested”, or “inadequate communication quality”), and classify terminal devices having communication characteristics CMbest into three groups CLS ₁ to CLS ₃ using the learned LDA algorithm.

More specifically, when "normal" is the result to be most accurately estimated, the estimation means 16 learns an LDA algorithm using scan data when the network scan is "normal" (communication characteristics CMbest when "normal"), and classifies terminal devices having communication characteristics CMbest into three groups _CLS1 to _CLS3 using the learned LDA algorithm.

Furthermore, when "network congested" is the result to be most accurately estimated, the estimation means 16 learns an LDA algorithm using scan data when the network scan results in "network congested" (communication characteristics CMbest when "network congested"), and classifies terminal devices having communication characteristics CMbest into three groups _CLS1 to _CLS3 using the learned LDA algorithm.

Furthermore, when "inadequate communication quality" is the result to be most accurately estimated, the estimation means 16 learns an LDA algorithm using scan data when the network scan results in "inadequate communication quality" (communication characteristics CMbest when the communication quality is "inadequate"), and classifies terminal devices having communication characteristics CMbest into three groups _CLS1 to _CLS3 using the learned LDA algorithm.

In this way, the estimation means 16 may determine the scan data that is the teacher data according to the result that is desired to be estimated most accurately, and may learn an LDA algorithm using the determined scan data, and may classify terminal devices having communication characteristics CMbest into three groups _CLS1 to _CLS3 using the learned LDA algorithm.

Also, in the above description, it has been explained that the network scan is performed at each scan timing t _S (or scan timing t' _S ), but this is not limited to the embodiment of the present invention, and the network scan may be performed in units of a scan period T _S _total, or may be performed over a plurality of scan timings within the scan period T _S _total. Therefore, in the embodiment of the present invention, it is sufficient that the communication characteristics acquired by executing the network scan over a plurality of scan timings are used to estimate whether the network scan is "normal" or "failed" by the above-mentioned method, and when it is estimated that the network scan is "failed", the cause of the "failure" of the network scan is further estimated.

Fig. 49 is a diagram showing the probability density function of scan response delay RTT _m . In Fig. 49, the vertical axis represents the probability density function of scan response delay RTT _m , and the horizontal axis represents scan response delay RTT _m . In Fig. 49, (a) represents the probability density function of scan response delay RTT _m acquired in a communication environment where the result of a network scan is "normal", (b) represents the probability density function of scan response delay RTT _m acquired in a communication environment where the result of a network scan is "inadequate communication quality", (c) represents the probability density function of scan response delay RTT _m acquired in a communication environment where the result of a network scan is "network congestion", and (d) represents the probability density function of scan response delay RTT _m acquired in a communication environment where the result of a network scan is "unknown".

Referring to FIG. 49 , the probability density function ((a)) of the scan response delay RTT _m obtained in a communication environment where the result of the network scan is “normal”, the probability density function ((b)) of the scan response delay RTT _m obtained in a communication environment where the result of the network scan is “inadequate communication quality”, and the probability density function ((c)) of the scan response delay RTT _m obtained in a communication environment where the result of the network scan is “network congestion” are all different from each other.

When the probability density function of the scan response delay RTT _m is the probability density function shown in FIG. 49(a), the sample data used to estimate the result of the network scan (="normal") is standard sample data D_standard_1. When the probability density function of the scan response delay RTT _m is the probability density function shown in FIG. 49(b), the sample data used to estimate the result of the network scan (="poor communication quality") is standard sample data D_standard_2. When the probability density function of the scan response delay RTT _m is the probability density function shown in FIG. 49(c), the sample data used to estimate the result of the network scan (="network congestion") is standard sample data D_standard_3.

And the probability density function of the scan response delay RTT _m shown in FIG. 49(d) is most similar to the probability density function of the scan response delay RTT _m shown in FIG. 49(a).

Therefore, it has been found that by detecting the probability density function of scan response delay RTT _m most similar to the probability density function of scan response delay RTT m shown in FIG. 49(d) from the three probability density functions of scan response delay RTT _m shown in FIG. 49(a) to (c) in accordance with steps S129 to S139 shown in FIGS _. 36 and 37, it is possible to accurately detect the probability density function of scan response delay RTT _m most similar to the probability density function of scan response delay RTT _m shown in FIG. 49(d).

Furthermore, if the probability density function of scan response delay RTT _m that is most similar to the probability density function of scan response delay RTT _m shown in FIG. 49(d) (= the probability density function of scan response delay RTT _m shown in FIG. 49(a)) is detected, the network scan result may be estimated to be “normal”. However, when the network scan result is estimated by the estimation algorithm ALG, the network scan result is estimated using communication characteristic CMbest _m in addition to the scan response delay RTT _m , so it cannot be concluded that the estimated network scan result will always be “normal”.

Therefore, the estimation algorithm ALG is trained using initial sample data D_sample_initial generated by combining standard sample data D_standard_1 used when the network scan result is estimated to be "normal", standard sample data D_standard_2 used when the network scan result is estimated to be "inadequate communication quality", and standard sample data D_standard_3 used when the network scan result is estimated to be "network congestion", and tentative sample data D_sample_tentative_1 to D_sample_tentative_U generated based on the initial sample data D_sample_initial, and the estimation algorithm ALG is trained using a validation set (=consisting of all of the subsets V _1' to V _U' ) and test data (=subsets TD ₁ to TD _4. ) is used to estimate the results of a network scan using an estimation algorithm ALG (trained estimation algorithm ALG), and the estimation results of the network scan are evaluated using evaluation indexes EVD, nEVD, and tEVD, thereby updating the estimation algorithm ALG so as to approach the optimal algorithm (see the flowcharts shown in FIGS. 36 to 38).

Therefore, by estimating the results of a network scan using the estimation algorithm ALG, which is close to the optimal algorithm, the results of a network scan can be estimated with the highest accuracy.

At the time when the execution of the flowcharts shown in Figures 36 to 38 is completed, the execution of the flowcharts shown in Figures 40 to 42 is completed in step S161 of Figure 38, so the result of the network scan is estimated to be either "normal", "inadequate communication quality", "network congestion", "port closed" or "no connection". If the estimated result of the network scan is "inadequate communication quality" or "network congestion", a rescan list is created in step S12 of Figure 33, and then a network scan is performed based on the rescan list in step S4. As a result, even if the estimated result of the network scan is "inadequate communication quality" or "network congestion", the estimated result of the network scan may become "normal" by rescanning, and the result of the network scan can be accurately estimated.

In an embodiment of the present invention, the estimated result of a network scan consisting of "normal" constitutes the "first estimated result", the estimated result of a network scan consisting of "inadequate communication quality" constitutes the "second estimated result", and the estimated result of a network scan consisting of "network congestion" constitutes the "third estimated result". The first estimated result, the second estimated result, and the third estimated result constitute the "multiple known estimated results".

In addition, in this embodiment of the present invention, the standard sample data D_standard_1 constitutes the "first standard sample data", the standard sample data D_standard_2 constitutes the "second standard sample data", and the standard sample data D_standard_3 constitutes the "third standard sample data".

Furthermore, in an embodiment of the present invention, the first estimation result may be a result estimated by the estimation algorithm ALG using standard sample data D_standard_1, the second estimation result may be a result estimated by the estimation algorithm ALG using standard sample data D_standard_2, and the third estimation result may be a result estimated by the estimation algorithm ALG using standard sample data D_standard_3. Also, each of the first estimation result, the second estimation result, and the third estimation result may be a result estimated by an algorithm other than the estimation algorithm ALG.

In the embodiment of the present invention, the threshold value _{Corr_r} constitutes a "first threshold value", the threshold value _{S_r} constitutes a "second threshold value", and the threshold value (N _terminal ×D_thres) constitutes a "third threshold value".

In addition, in this embodiment of the present invention, the communication characteristic CMbest constitutes a "first communication characteristic", the communication characteristics having a correlation coefficient r _ij greater than the threshold value Corr _r constitute "D (D is a positive integer) second communication characteristics", and the communication characteristics extracted in descending order of the score Q constitute "K (K is a positive integer) second communication characteristics".

Furthermore, in an embodiment of the present invention, the D score or count value DS_count constitutes a "discrimination score."

Furthermore, in this embodiment of the invention, the D score D_2 constitutes the "first discrimination score", the D score D_3 constitutes the "second discrimination score", and the D score D_1 constitutes the "third discrimination score".

Furthermore, in this embodiment of the present invention, the packet loss rate PLR _m , jitter JT _m , scanning maximum delay DLmax _m , and scanning minimum delay DLmin _m shown in FIG. 13 constitute "D (D is a positive integer) second communication characteristics" whose correlation coefficient r _ij is greater than a threshold value Corr _r .

Furthermore, in an embodiment of the present invention, the K communication characteristics described above constitute "K second communication characteristics."

Furthermore, in this embodiment of the present invention, group CLS ₁ constitutes a "first group" in which the network scan is normal, and groups CLS ₂ and CLS ₃ constitute a "second group" and a "third group" respectively in which the network scan has failed due to a cause attributable to the communication channel with the terminal device.

Furthermore, in this embodiment of the invention, Model 2 constitutes the "first model" and Model 3 constitutes the "second model."

The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is indicated by the claims, not by the description of the embodiments above, and is intended to include all modifications within the meaning and scope of the claims.

This invention applies to a network scanning device, a program for causing a computer to execute the program, and a computer-readable recording medium on which the program is recorded.

1 Network scanning device, 10 Communication system, 11 Network scanner, 12 Scan analysis manager, 13 Host information database, 14 Extraction means, 15 Calculation means, 16 Estimation means, 17 Scan scheduler, 20 Scan target network, 30 User.

Claims (19)

a scanning means for executing a network scan in a first communication environment in which the results of a network scan are unknown for M (M is an integer equal to or greater than 2) terminal devices, and acquiring a first scan response delay of the network scan for each of the M terminal devices;
a detection means for executing a detection process for detecting, as a similar scan response delay, M' second scan response delays that are most similar to the M first scan response delays among a plurality of sets of M'(M' is an integer equal to or greater than 2) second scan response delays obtained when the results of a network scan executed in a second communication environment in which the results of the network scan are known are a plurality of known estimated results;
a generating means for acquiring, for each of the M' second scan response delays, an optimal communication characteristic that is the communication characteristic most suitable for estimating the known estimation result among a plurality of communication characteristics acquired together with one of the M' second scan response delays constituting the similar scan response delay, and generating similar standard sample data consisting of the acquired M' optimal communication characteristics and the M' second scan response delays;
an update means for training an estimation algorithm for estimating the result of the network scan executed in the first communication environment based on the similar standard sample data, and updating the estimation algorithm using an evaluation index so as to approach an optimal algorithm;
an estimation means for estimating a result of a network scan executed in the first communication environment by using the estimation algorithm updated by the update means,
the optimization algorithm comprises an algorithm for estimating a result of a network scan such that the estimated result of the network scan matches the known estimated result;
A network scanning device, wherein the evaluation index is an index for evaluating the accuracy of an estimation result when the result of a network scan is estimated using the trained algorithm. The generating means further generates training data and test data from the similar standard sample data, and generates a training set and a validation set from the training data, generates initial sample data by combining a portion of each of a plurality of standard sample data corresponding to the plurality of known estimation results and each of which is composed of the M′ second scan response delays and the M′ optimal communication characteristics, and generates provisional sample data by replacing a portion of the initial sample data composed of standard sample data of the same type as the training set with a portion of the training set;
2. The network scanning device according to claim 1, wherein the update means updates the estimation algorithm with the provisional sample data when a first evaluation index for evaluating an estimation result when the estimation algorithm is trained using the initial sample data and the network scan result is estimated by the trained estimation algorithm using the validation set is equal to or less than a second evaluation index for evaluating an estimation result when the estimation algorithm is trained using the provisional sample data and the network scan result is estimated by the trained estimation algorithm using the validation set. the plurality of known inference results include a first inference result indicating that the result of the network scan is normal, a second inference result indicating that the network scan has failed due to insufficient communication quality, and a third inference result indicating that the network scan has failed due to network congestion;
The plurality of standard sample data are
first standard sample data, which is standard sample data for estimating that the result of the network scan is the first estimation result, and which is composed of the M' second scan response delays and the M' optimal communication characteristics acquired in a communication environment in which a network is normal;
second standard sample data, which is standard sample data for estimating that the result of the network scan is the second estimation result, and which is composed of the M' second scan response delays and the M' optimal communication characteristics acquired in a communication environment in which network communication quality is insufficient;
and third standard sample data, which is standard sample data for estimating that the result of the network scan is the third estimation result, the third standard sample data being composed of the M' second scan response delays and the M' optimal communication characteristics acquired in a communication environment in which a network is congested,
the first estimation result is a result estimated by the estimation algorithm using the first standard sample data;
the second estimation result is a result estimated by the estimation algorithm using the second standard sample data;
3. The network scanning device according to claim 2, wherein the third estimation result is a result estimated by the estimation algorithm using the third standard sample data. the generating means replaces a part of the sample data, which is made of standard sample data of the same type as the training set, among the initial sample data with one of a plurality of subset data of the training set, for all of the plurality of subset data, to generate a plurality of provisional sample data constituting the provisional sample data;
The update means includes a first process for training the estimation algorithm using the initial sample data, and generating a first confusion matrix based on an estimation result obtained when a result of a network scan is estimated by the trained estimation algorithm using the validation set; a second process for training the estimation algorithm using one of a plurality of provisional sample data constituting the provisional sample data, and generating a second confusion matrix based on an estimation result obtained when a result of a network scan is estimated by the trained estimation algorithm using the validation set, for all of the plurality of provisional sample data; and a first evaluation index calculated based on the first confusion matrix and a second confusion matrix calculated based on the plurality of second confusion matrices. 4. The network scanning device according to claim 2, further comprising: a third process for updating the estimation algorithm with optimal sample data consisting of one provisional sample data among the plurality of provisional sample data when a maximum evaluation index of the plurality of second evaluation indexes is obtained, the optimal sample data being greater than the first evaluation index and being based on a plurality of second evaluation indexes obtained by the estimation; a fourth process for generating a third confusion matrix based on an estimation result obtained when a result of a network scan is estimated by the updated estimation algorithm using the test data after the third process; and a fifth process for approving the updated estimation algorithm and the optimal sample data when a third evaluation index calculated based on the third confusion matrix is equal to or greater than the maximum evaluation index. The network scanning device according to claim 4, wherein the update means, when the third evaluation index is smaller than the maximum evaluation index, repeatedly executes the first process to the fifth process using the estimation algorithm updated with the optimal sample data until the third evaluation index becomes equal to or greater than the maximum evaluation index. Further comprising a calculation means for calculating the evaluation index,
Each of the first to third confusion matrices includes a first element indicating a rate at which an assumed event is correctly estimated when the assumed event occurs, a second element indicating a rate at which an assumed event is correctly estimated not to have occurred when the assumed event does not actually occur, a third element indicating a rate at which an assumed event is erroneously estimated to have occurred as another event when the assumed event occurs, and a fourth element indicating a rate at which an assumed event is erroneously estimated to have occurred when the assumed event does not actually occur;
6. The network scanning device according to claim 4, wherein the calculation means calculates Precision, which is a ratio of a ratio actually classified into the first element among a ratio classified into the first element or the fourth element, and Recall, which is a ratio of a ratio actually classified into the first element among a ratio classified into the first element or the third element, and calculates 2×Precision/(Precision+Recall) based on the Precision and the Recall, based on the first confusion matrix to calculate the first evaluation index, executes the third calculation process based on the plurality of second confusion matrices to calculate the plurality of second evaluation indexes, and executes the third calculation process based on the third confusion matrix to calculate the third evaluation index. Further comprising a calculation means for calculating the evaluation index,
Each of the first to third confusion matrices includes a first element indicating a rate at which an assumed event is correctly estimated when the assumed event occurs, a second element indicating a rate at which an assumed event is correctly estimated not to have occurred when the assumed event does not actually occur, a third element indicating a rate at which an assumed event is erroneously estimated to have occurred as another event when the assumed event occurs, and a fourth element indicating a rate at which an assumed event is erroneously estimated to have occurred when the assumed event does not actually occur;
6. The network scanning device according to claim 4, wherein the calculation means calculates the first evaluation index consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the first confusion matrix, calculates the plurality of second evaluation indexes consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the plurality of second confusion matrices, and calculates the third evaluation index consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the third confusion matrix. The network scanning device according to any one of claims 1 to 7, wherein the detection means detects, in the detection process, the M' second scan response delays having a distribution most similar to the distribution of the M first scan response delays from the multiple sets of M' second scan response delays as the similar scan response delay based on the multiple distributions of the multiple sets of M' second scan response delays and the distribution of the M first scan response delays. The network scanning device according to claim 8, wherein the detection means performs a first calculation process to calculate a first mode indicating the most likely value of the first scan response delay based on the M first scan response delays, and calculate a first mean absolute deviation that is the average value of the absolute value of the difference between the M first scan response delays and the first mode, and after the first calculation process, calculate a second mode indicating the most likely value of the second scan response delay based on the M' second scan response delays, calculate a second mean absolute deviation that is the average value of the absolute value of the difference between the M' second scan response delays and the second mode, calculate a difference in mean absolute deviation consisting of the absolute value of the difference between the first mean absolute deviation and the second mean absolute deviation, and subtract the difference in mean absolute deviation from 1 to calculate a similarity, for all of the multiple known estimation results to obtain multiple similarities, and detects the M' second scan response delays that were the basis for calculating the highest similarity among the multiple obtained similarities as the similar scan response delay. A first step in which a scanning means executes a network scan in a first communication environment in which the results of a network scan are unknown for M (M is an integer equal to or greater than 2) terminal devices, and acquires a first scan response delay of the network scan for each of the M terminal devices;
a second step of executing a detection process in which the detection means detects, as similar scan response delays, M' second scan response delays that are most similar to the M first scan response delays among a plurality of sets of M'(M' is an integer equal to or greater than 2) second scan response delays obtained when the results of a network scan executed in a second communication environment in which the results of the network scan are known are a plurality of known estimated results;
a third step in which a generating means acquires, for each of the M' second scan response delays, an optimal communication characteristic that is the communication characteristic most suitable for estimating the known estimation result among a plurality of communication characteristics acquired together with one of the M' second scan response delays constituting the similar scan response delay, and generates similar standard sample data consisting of the acquired M' optimal communication characteristics and the M' second scan response delays;
a fourth step in which an update means trains an estimation algorithm for estimating a result of a network scan executed in the first communication environment based on the similar standard sample data, and updates the estimation algorithm using an evaluation index so as to approach an optimal algorithm;
a fifth step in which an estimation means estimates a result of a network scan performed in the first communication environment using the estimation algorithm updated in the fourth step;
the optimization algorithm comprises an algorithm for estimating a result of a network scan such that the estimated result of the network scan matches the known estimated result;
The evaluation index is an index for evaluating the accuracy of an estimation result when the result of a network scan is estimated by the trained algorithm. In the third step, the generating means further generates training data and test data from the similar standard sample data, and generates a training set and a validation set from the training data, generates initial sample data by combining a portion of a plurality of standard sample data that correspond to the plurality of known estimation results and each of which is composed of the M′ second scan response delays and the M′ optimal communication characteristics, and generates temporary sample data by replacing a portion of the initial sample data composed of standard sample data of the same type as the training set with a portion of the training set,
11. The program for causing a computer to execute the program according to claim 10, wherein in the fourth step, when a first evaluation index for evaluating an estimation result obtained when the estimation algorithm is trained using the initial sample data and the network scan result is estimated by the trained estimation algorithm using the validation set is equal to or lower than a second evaluation index for evaluating an estimation result obtained when the estimation algorithm is trained using the provisional sample data and the network scan result is estimated by the trained estimation algorithm using the validation set, the update means updates the estimation algorithm with the provisional sample data. the plurality of known inference results include a first inference result indicating that the result of the network scan is normal, a second inference result indicating that the network scan has failed due to insufficient communication quality, and a third inference result indicating that the network scan has failed due to network congestion;
The plurality of standard sample data are
first standard sample data, which is standard sample data for estimating that the result of the network scan is the first estimation result, and which is composed of the M' second scan response delays and the M' optimal communication characteristics acquired in a communication environment in which a network is normal;
second standard sample data, which is standard sample data for estimating that the result of the network scan is the second estimation result, and which is composed of the M' second scan response delays and the M' optimal communication characteristics acquired in a communication environment in which network communication quality is insufficient;
and third standard sample data, which is standard sample data for estimating that the result of the network scan is the third estimation result, the third standard sample data being composed of the M' second scan response delays and the M' optimal communication characteristics acquired in a communication environment in which a network is congested,
the first estimation result is a result estimated by the estimation algorithm using the first standard sample data;
the second estimation result is a result estimated by the estimation algorithm using the second standard sample data,
The program for causing a computer to execute the program according to claim 11 , wherein the third estimation result is a result estimated by the estimation algorithm using the third standard sample data. In the third step, the generating means replaces a part of the sample data of the initial sample data, which is made of standard sample data of the same type as the training set, with one of a plurality of subset data of the training set, for all of the plurality of subset data, to generate a plurality of provisional sample data constituting the provisional sample data;
The fourth step includes:
a first sub-step in which the update means trains the estimation algorithm using the initial sample data, and generates a first confusion matrix based on an estimation result when the trained estimation algorithm estimates a result of a network scan using the validation set;
a second sub-step in which the update means trains the estimation algorithm using a first set of provisional sample data among a plurality of provisional sample data constituting the provisional sample data, and generates a second confusion matrix based on an estimation result obtained when a result of a network scan is estimated by the trained estimation algorithm using the validation set;
a third sub-step of the update means setting the second evaluation index to the first evaluation index when the second evaluation index calculated based on the second confusion matrix is greater than the first evaluation index calculated based on the first confusion matrix, and updating the estimation algorithm with the first provisional sample data;
a fourth sub-step in which the update means does not set the second evaluation index to the first evaluation index when the second evaluation index is not greater than the first evaluation index, and selects second provisional sample data different from the first provisional sample data from a plurality of provisional sample data;
a fifth substep in which the updating means repeatedly executes the second substep to the fourth substep for all of the plurality of virtual sample data;
a sixth sub-step of generating a third confusion matrix based on an estimation result when the update means estimates a result of a network scan by the estimation algorithm finally updated in the fifth sub-step using the test data;
13. The program for causing a computer to execute the program according to claim 11 or 12, further comprising a seventh substep in which the updating means approves the finally updated estimation algorithm when a third evaluation index calculated based on the third confusion matrix is equal to or greater than the first evaluation index finally obtained in the fifth substep, and the first provisional sample data when the second evaluation index constituting the finally obtained first evaluation index is obtained. The fourth step includes:
14. The program for causing a computer to execute the program according to claim 13, further comprising an eighth substep in which, when the third evaluation index is smaller than the first evaluation index finally obtained in the fifth substep, the updating means repeatedly executes the second substep to the seventh substep using the finally updated estimation algorithm until the third evaluation index becomes equal to or larger than the first evaluation index finally obtained in the fifth substep. The calculation means further causes the computer to execute a sixth step of calculating the evaluation index;
Each of the first to third confusion matrices includes a first element indicating a rate at which an assumed event is correctly estimated when the assumed event occurs, a second element indicating a rate at which an assumed event is correctly estimated not to have occurred when the assumed event does not actually occur, a third element indicating a rate at which an assumed event is erroneously estimated to have occurred as another event when the assumed event occurs, and a fourth element indicating a rate at which an assumed event is erroneously estimated to have occurred when the assumed event does not actually occur;
The calculation means, in the sixth step, calculates Precision, which is a ratio of the ratio of the ratio classified into the first element or the fourth element to the ratio actually classified into the first element, and Recall, which is a ratio of the ratio of the ratio classified into the first element or the third element to the ratio actually classified into the first element, and calculates 2×Precision/(Precision+Recall) based on the Precision and the Recall based on the first confusion matrix to calculate the first evaluation index, and calculates the multiple second confusion matrices to calculate the multiple second evaluation indexes, and calculates the third evaluation index based on the third confusion matrix. The program for causing a computer to execute the program according to claim 13 or 14. The calculation means further causes the computer to execute a sixth step of calculating the evaluation index;
Each of the first to third confusion matrices includes a first element indicating a rate at which an assumed event is correctly estimated when the assumed event occurs, a second element indicating a rate at which an assumed event is correctly estimated not to have occurred when the assumed event does not actually occur, a third element indicating a rate at which an assumed event is erroneously estimated to have occurred as another event when the assumed event occurs, and a fourth element indicating a rate at which an assumed event is erroneously estimated to have occurred when the assumed event does not actually occur;
15. The program for causing a computer to execute the program according to claim 13 or 14, wherein, in the sixth step, the calculation means calculates the first evaluation index consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the first confusion matrix, calculates the plurality of second evaluation indexes consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the plurality of second confusion matrices, and calculates the third evaluation index consisting of a Matthews correlation coefficient using the first element, the second element, the third element, and the fourth element of the third confusion matrix. The program for causing a computer to execute the program according to any one of claims 10 to 16, wherein the detection means detects, in the detection process of the second step, the M' second scan response delays having a distribution most similar to the distribution of the M first scan response delays from the multiple sets of M' second scan response delays based on the multiple distributions of the multiple sets of M' second scan response delays and the distribution of the M first scan response delays, as the similar scan response delay. In the detection process of the second step, the detection means calculates a first mode indicating the value of the first scan response delay most likely to occur based on the M first scan response delays, and calculates a first mean absolute deviation, which is the average value of the absolute values of the differences between the M first scan response delays and the first mode; and after the first calculation process, calculates a second mode indicating the value of the second scan response delay most likely to occur based on the M' second scan response delays, and calculates a first mean absolute deviation, which is the average value of the absolute values of the differences between the M' second scan response delays and the second mode. 18. The program for causing a computer to execute the program according to claim 17, which calculates a second mean absolute deviation, which is the average value of the absolute values of the differences between the first mean absolute deviation and the second mean absolute deviation, calculates a difference in mean absolute deviation consisting of the absolute values of the differences between the first mean absolute deviation and the second mean absolute deviation, and subtracts the difference in mean absolute deviation from 1 to calculate the similarity, and executes a second calculation process for all of the multiple known estimation results to obtain multiple similarities, and detects the M' second scan response delays, which were used to calculate the highest similarity among the multiple similarities obtained, as the similar scan response delay. A computer-readable recording medium having recorded thereon the program according to any one of claims 10 to 18.

Images