JP7658466B2 - Information processing device, information processing system, customer device, and information processing program - Google Patents

Description

The present disclosure relates to an information processing device, an information processing system, a customer device, an information processing method, and a non-transitory computer-readable medium.

Patent document 1 describes a vibration measuring device for measuring the vibration of an object such as a mechanical facility. The vibration measuring device of Patent document 1 processes time-series data of the vibration of the object as a processed image.

Patent Document 2 describes an analysis device that analyzes the relationships between multiple pieces of time-series data from industrial machinery, etc. The analysis device of Patent Document 2 analyzes the change trends of section data separated by a sliding window width adjustment unit at the separation positions of the time-series data.

Patent Document 3 describes a failure solution prediction system that generates a solution when an error occurs. The failure solution prediction system of Patent Document 3 generates a solution based on the processing performed to deal with the failure.

Patent Document 4 describes a fault/fault determination device that classifies data obtained from multiple sensors by organizing them into a time series for each predetermined time unit, and determines whether a fault has occurred. The fault/fault determination device of Patent Document 4 classifies data by performing deep learning classification on graph data that has been graphed for each of the multiple sensors.

Patent document 5 describes an information processing system that performs failure prediction based on historical information acquired from equipment in a customer environment. The information processing system of Patent document 5 analyzes historical information of equipment acquired from the customer environment, and performs failure prediction by analyzing the degree of influence of each item of the historical information on a failure that occurs in the equipment.

Patent Document 6 describes an anomaly diagnosis method for diagnosing anomalies in equipment to be diagnosed. The anomaly diagnosis method of Patent Document 6 diagnoses an anomaly in the operating state of the equipment using multivariate time series data for a plurality of measurement items, and when an anomaly in the operating state of the equipment is determined, diagnoses the measurement item causing the anomaly from the difference between a feature value extracted from the multivariate time series data of a first section before the occurrence of the anomaly and a feature value extracted from the multivariate time series data of a second section after the occurrence of the anomaly.

International Publication No. 2020/255728 JP 2021-076998 A JP 2019-153306 A JP 2018-120487 A JP 2017-027124 A Patent No. 6851558

Related technologies for predicting failures include those that search for similar patterns in data and predict failures etc. when data similar to past failure data is found. However, such technologies may have a fixed cut-out period for pattern extraction. As a result, it may not be possible to detect the characteristics of the failure data unless the range of pattern extraction is expanded, making it difficult to predict and determine failures with high accuracy.

In view of the above-mentioned problems, the object of the present disclosure is to provide an information processing device, an information processing system, a customer device, an information processing method, and a non-transitory computer-readable medium that can determine a fault with high accuracy.

An information processing device according to one aspect of the present disclosure includes an image data acquisition means for acquiring failure representative image data that is an image of a portion of time series data of sensor values acquired by a sensor device of a customer device when the customer device is broken, a countermeasure acquisition means for acquiring countermeasures at the time of the failure, an image data conversion means for converting a portion of the time series data of the sensor values acquired by the sensor device of the customer device that is the subject of a failure judgment into target image data, an image data determination means for determining a failure of the customer device by comparing the failure representative image data with the target image data, and an image data determination means for determining the failure of the customer device by comparing the failure representative image data with the target image data when the failure is judged to be a failure. and a countermeasure notification means for notifying a customer device, wherein the image data conversion means converts the target image data to include a plurality of first target image data obtained by converting a first window frame, which divides the time series data with a time width of a first time, into an image from each first window frame while going back along the time axis of the time series data to a predetermined time at a predetermined first going back time interval, and a plurality of second target image data obtained by converting a second window frame, which divides the time series data with the time width of a second time larger than the first time, into an image from each second window frame while going back along the time axis of the time series data to the predetermined time at a predetermined second going back time interval.

An information processing system according to one aspect of the present disclosure includes a customer device including a sensor device that acquires sensor values, an information processing device connected to the customer device via a network, a failure representative image data storage device that stores failure representative image data that is an image of a portion of the time series data of the sensor values acquired by the sensor device of the customer device when the customer device fails, and a countermeasure storage device that stores countermeasures to be taken at the time of the failure, and the information processing device includes a failure representative image data acquisition means that acquires the failure representative image data, a countermeasure acquisition means that acquires the countermeasures to be taken at the time of the failure, an image data conversion means that converts a portion of the time series data of the sensor values acquired by the sensor device of the customer device that is the subject of failure judgment into target image data, and the failure representative image data. The system has an image data determination means for determining a malfunction of the customer device by comparing it with the target image data, and a countermeasure notification means for notifying the customer device of the countermeasure when a malfunction is determined, and the image data conversion means converts the target image data to include a plurality of first target image data obtained by converting a first window frame, which divides the time series data with a time width of a first time, into an image from each first window frame while moving back along the time axis of the time series data to a predetermined time at a predetermined first going back time interval, and a plurality of second target image data obtained by converting a second window frame, which divides the time series data with a time width of a second time greater than the first time, into an image from each second window frame while moving back along the time axis of the time series data to the predetermined time at a predetermined second going back time interval.

A customer device according to one aspect of the present disclosure includes a sensor device for acquiring sensor values, and a display means for displaying countermeasures to be taken when a failure is determined, and includes image data acquisition means for acquiring representative failure image data that is an image of a portion of the time series data of the sensor values acquired by the sensor device of the device when the device is broken, a countermeasure acquisition means for acquiring countermeasures to be taken when the device is broken, an image data conversion means for converting a portion of the time series data of the sensor values acquired by the sensor device of the device that is the subject of failure judgment into target image data, an image data determination means for determining a failure of the device by comparing the representative failure image data with the target image data, and a display means for displaying countermeasures to be taken when a failure is determined. and a countermeasure notification means for notifying the device of a countermeasure to be taken. The image data conversion means is connected via a network to an information processing device having the image data conversion means for converting the image data into the target image data including: a first window frame that divides the time series data with a time width of a first time, while moving the data back along the time axis of the time series data to a predetermined time at a predetermined first going back time interval, into a plurality of first target image data pieces that are converted into images from each first window frame; and a second window frame that divides the time series data with the time width of a second time larger than the first time, while moving the data back along the time axis of the time series data to the predetermined time at a predetermined second going back time interval.

An information processing method according to one aspect of the present disclosure includes an image data acquisition step of acquiring failure representative image data that is an image of a portion of time series data of sensor values acquired by a sensor device of a customer device when the customer device is broken, a countermeasure acquisition step of acquiring a countermeasure at the time of the failure, an image data conversion step of converting a portion of the time series data of the sensor values acquired by the sensor device of the customer device that is the subject of a failure determination into target image data, an image data determination step of determining a failure of the customer device by comparing the failure representative image data with the target image data, and, when the failure is determined, acquiring the countermeasure. and a countermeasure notification step of notifying the customer device of the countermeasure. In the image data conversion step, the target image data is converted to include a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data with a time width of a first time, into an image from each first window frame while going back along the time axis of the time series data to a predetermined time at a predetermined first going back time interval, and a plurality of second target image data pieces obtained by converting a second window frame, which divides the time series data with the time width of a second time larger than the first time, into an image from each second window frame while going back along the time axis of the time series data to the predetermined time at a predetermined second going back time interval.

A non-transitory computer-readable medium according to one aspect of the present disclosure includes an image data acquisition step for acquiring representative failure image data that is an image of a portion of time-series data of sensor values acquired by a sensor device of a customer device when the customer device is broken; a countermeasure acquisition step for acquiring a countermeasure at the time of the failure; an image data conversion step for converting a portion of the time-series data of the sensor values acquired by the sensor device of the customer device that is the subject of a failure determination into target image data; an image data determination step for determining a failure of the customer device by comparing the representative failure image data with the target image data; and a countermeasure notification step for notifying the customer device of the countermeasure when the failure is determined. The information processing program stored in the computer causes the computer to execute the above-mentioned detection step and the image data conversion step to convert the target image data to include a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data with a time width of a first time, into images from each first window frame while moving the data back along the time axis of the time series data to a predetermined time at a predetermined first going back time interval, and a plurality of second target image data pieces obtained by converting a second window frame, which divides the time series data with the time width of a second time greater than the first time, into images from each second window frame while moving the data back along the time axis of the time series data to the predetermined time at a predetermined second going back time interval.

The present disclosure makes it possible to provide an information processing device, an information processing system, a customer device, an information processing method, and a non-transitory computer-readable medium that can determine faults with high accuracy.

1 is a configuration diagram illustrating an information processing system according to a first embodiment. 1 is a graph showing an image of time-series data of sensor values acquired by a sensor device of a customer apparatus according to the first embodiment, in which the horizontal axis indicates time and the vertical axis indicates intensity. 4 is a diagram illustrating an example of failure representative image data stored in a failure representative image data storage device according to the first embodiment; FIG. 1 is a block diagram illustrating an information processing apparatus according to a first embodiment. 5 is a diagram illustrating an example of target image data converted by an image data conversion unit in the information processing device according to the first embodiment. 5 is a diagram illustrating an example of target image data converted by an image data conversion unit in the information processing device according to the first embodiment. FIG. FIG. 2 is a flowchart illustrating an information processing method according to the first embodiment. FIG. 11 is a configuration diagram illustrating an information processing system according to a second embodiment. FIG. 11 is a block diagram illustrating an information processing device according to a second embodiment. 11 is a diagram illustrating an example of a relationship between each component of an information processing system according to a second embodiment and each component of an information processing device. 13 is a flowchart illustrating a processing method of an image data determination unit, a countermeasure notification unit, and an image data vectorization unit in an information processing device according to a second embodiment. 13 is a diagram illustrating an example of failure representative image data extracted by an image data extracting unit in the information processing device according to the second embodiment. FIG. 13A to 13C are diagrams illustrating examples of failure representative image data extracted by an image data extracting unit with different time widths in an information processing device according to a second embodiment. FIG. 11 is a flowchart illustrating a method for extracting failure representative image data in the information processing device according to the second embodiment. FIG. 11 is a configuration diagram illustrating an information processing system according to a third embodiment. FIG. 11 is a block diagram illustrating an information processing device according to a third embodiment. FIG. 11 is a flowchart illustrating a method for processing simulated fault image data in an information processing device according to a third embodiment. FIG. 13 is a block diagram illustrating an example of a customer device to which data is uploaded in an information processing system according to a fourth embodiment. FIG. 13 is a block diagram illustrating a customer device that is not a target for uploading in an information processing system according to a fourth embodiment.

Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and duplicated explanations will be omitted as necessary for clarity of explanation.

(Embodiment 1)
An information processing system and an information processing device according to the first embodiment will be described. Fig. 1 is a configuration diagram illustrating an information processing system according to the first embodiment. As shown in Fig. 1, the information processing system 1 includes a customer device M01, etc., an information processing device 100, a representative failure image data storage device 10, and a countermeasure storage device 20. The customer device M01, etc., and the information processing device 100 function as information processing means. The representative failure image data storage device 10 and the countermeasure storage device 20 function as representative failure image data storage means and countermeasure storage means, respectively.

The information processing system 1 may include customer devices M01, M02, and M03. Customer devices M01, M02, and M03 are collectively referred to as customer device M0. Although three customer devices M0 are shown in the figure, the number of customer devices M0 is not limited to three, and may be one to two, or four or more. Customer device M0 is connected to the information processing device 100 via the network NW in a communicable state.

The customer device M0 is equipped with various sensor devices SD. The various sensor devices SD monitor the status of each part and component of the customer device M0 in which they are installed. For example, the various sensor devices SD monitor temperature, humidity, voltage, vibration, etc. as the status of each part and component of the customer device M0.

Figure 2 is a graph showing the time series data of the sensor values acquired by the sensor device SD of the customer device M0 in embodiment 1, where the horizontal axis indicates time and the vertical axis indicates intensity. As shown in Figure 2, each sensor device SD of the customer devices M01, M02, and M03 acquires time series data 31 of the sensor values of elements A to C. Elements A to C indicate, for example, temperature, humidity, voltage, vibration, etc. In this way, the customer device M0 includes a sensor device SD that acquires a sensor value. The customer device M0 may include multiple sensor devices SD that acquire multiple different types of sensor values. The sensor value may include a first sensor value acquired by a first sensor device SD and a second sensor value acquired by a second sensor device SD different from the first sensor device SD. The customer device M0 outputs the time series data 31 of the acquired sensor values to the information processing device 100 via the network NW.

The network NW may include a wireless notification line or a wired network line. The network NW may combine wireless and wired communication lines. The network NW may constitute a cloud.

The failure representative image data storage device 10 is connected to the information processing device 100 in a communicable state. FIG. 3 is a diagram illustrating an example of failure representative image data stored in the failure representative image data storage device 10 in embodiment 1. As shown in FIG. 3, the failure representative image data storage device 10 stores failure representative image data 11 and 12. The failure representative image data 11 and 12 are images of a portion of the time series data 31 of the sensor value acquired by the sensor device SD of the customer device M0 when the customer device M0 fails. The failure representative image data 11 and 12 include time series data 31 at the time of the failure and immediately before the failure.

The time width of the failure representative image data 12 is, for example, larger than the time width of the failure representative image data 11. For example, the failure representative image data 11 includes feature 31a as a feature of the time series data 31. On the other hand, the failure representative image data 12 includes feature 31b in addition to feature 31a as a feature of the time series data 31. Therefore, if the cause of the failure is in feature 31b, the failure representative image data 12 can detect the cause of the failure. The failure representative image data storage device 10 outputs the failure representative image data 11 and 12 to the information processing device 100.

The countermeasure storage device 20 is connected to the information processing device 100 in a communicable state. The countermeasure storage device 20 stores countermeasures in the event of a failure of the customer device M0. The countermeasures may include the cause and reason for the failure of the customer device M0, or may include countermeasures used in the past. The countermeasures may also include countermeasures for each pattern of the representative failure image data 11 and 12, or may include countermeasures for each of the elements A to C. The countermeasure storage device 20 may store various information when the customer device M0 fails as countermeasures. The countermeasure storage device 20 outputs countermeasures corresponding to the representative failure image data 11 and 12 to the information processing device 100.

The information processing device 100 is connected to the customer device M0 via a network NW. The information processing device 100 is also connected to a failure representative image data storage device 10. The information processing device 100 is further connected to a countermeasure storage device 20.

Figure 4 is a block diagram illustrating an information processing device according to embodiment 1. As shown in Figure 4, the information processing device 100 includes an image data acquisition unit 110, a countermeasure acquisition unit 120, an image data conversion unit 130, an image data determination unit 140, and a countermeasure notification unit 150. The image data acquisition unit 110, the countermeasure acquisition unit 120, the image data conversion unit 130, the image data determination unit 140, and the countermeasure notification unit 150 each have the functions of image data acquisition means, countermeasure acquisition means, image data conversion means, image data determination means, and countermeasure notification means.

The image data acquisition unit 110 acquires failure representative image data such as failure representative image data 11 and 12 from the failure representative image data storage device 10.

The countermeasure acquisition unit 120 acquires countermeasures in the event of a failure from the countermeasure storage device 20. The countermeasure acquisition unit 120 may acquire a first countermeasure when a failure is determined in the time series data 31 of the first sensor value. The countermeasure acquisition unit 120 may also acquire a second countermeasure when a failure is determined in the time series data 31 of the second sensor value. Furthermore, the countermeasure acquisition unit 120 may acquire a third countermeasure when a failure is determined in the time series data 31 of the first sensor value and the second sensor value. That is, when a failure is determined in the time series data 31 of the first sensor value and the second sensor value, not only may the first countermeasure and the second countermeasure be acquired in a superimposed manner, but also a third countermeasure different from the first countermeasure and the second countermeasure may be acquired.

The image data conversion unit 130 converts a portion of the time series data 31 of the sensor value acquired by the sensor device SD of the customer device M0 that is the subject of the fault judgment into target image data. Figures 5 and 6 are diagrams illustrating examples of target image data converted by the image data conversion unit 130 in the information processing device 100 of embodiment 1.

As shown in Fig. 5, the image data conversion unit 130 sets a first window frame F1 that divides the time series data with a time width of a first time. The image data conversion unit 130 moves the set first window frame F1 back along the time axis of the time series data 31 to a predetermined time at a predetermined first back time interval. Then, while moving the first window frame F1 back, the image data conversion unit 130 generates a plurality of first target image data G11 to G13 that are converted into images from each first window frame F1.

As shown in FIG. 6, the image data conversion unit 130 sets a second window frame F2 that divides the time series data 31 into a second time that is larger than the first time. The image data conversion unit 130 moves the set second window frame F2 back along the time axis of the time series data 31 to a predetermined time at a predetermined second retroactive time interval. Then, while moving the second window frame F2 back, the image data conversion unit 130 generates a plurality of second target image data G21-G22 that are converted into images from each second window frame F2. It is desirable that the second retroactive time interval be larger than the first retroactive time interval. In this way, the image data conversion unit 130 converts a portion of the time series data 31 into target image data that includes the first target image data G11-G13 and the second target image data G21-G22.

The image data conversion unit 130 may generate multiple target image data by setting multiple window frames with further varying time widths, in addition to the first target image data and the second target image data. That is, after operating by taking a second window frame F2 with a larger time width than the first window frame F1, it may operate by taking a third window frame with a larger time width than the second window frame F2. This operation may also be continued. This operation may be terminated when the window frame width reaches a predetermined maximum window frame width or more.

The image data determination unit 140 determines a failure of the customer device M0 by comparing the failure representative image data stored in the failure representative image data storage device 10 with the target image data converted by the image data conversion unit 130. The image data determination unit 140 may compare the target image data obtained by imaging a portion of the time series data 31 of the first sensor value with the failure representative image data obtained by imaging a portion of the time series data 31 of the first sensor value. The image data determination unit 140 may also compare the target image data obtained by imaging a portion of the time series data 31 of the second sensor value with the failure representative image data obtained by imaging a portion of the time series data 31 of the second sensor value.

When the image data determination unit 140 determines that there is a failure, the countermeasure notification unit 150 notifies the customer device M0 of the countermeasure stored in the countermeasure storage device 20 via the network NW. For example, the countermeasure notification unit 150 may notify the customer device M0 of the first countermeasure when a failure is determined in the time series data 31 of the first sensor value. The countermeasure notification unit 150 may also notify the customer device M0 of the second countermeasure when a failure is determined in the time series data 31 of the second sensor value. Furthermore, the countermeasure notification unit 150 may notify the customer device M0 of the third countermeasure when a failure is determined in the time series data 31 of the first sensor value and the second sensor value. The countermeasure notification unit 150 may also notify the customer device M0 of past countermeasures for each similar pattern of the representative failure image data.

The information processing device 100 described above is, for example, an information processing device such as a server. Note that the information processing device 100 is not limited to a server, and may be a personal computer, tablet, mobile terminal, etc., as long as it performs information processing. The information processing device 100 may include a processor, memory, storage device, and communication device as components not shown. The storage device may also store the processing performed by each component of the information processing device 100 as a program. The processor may also load the program from the storage device into the memory and execute the program. In this way, the processor realizes the functions of the image data acquisition unit 110, the countermeasure acquisition unit 120, the image data conversion unit 130, the image data determination unit 140, and the countermeasure notification unit 150.

Each of the components of the information processing device 100 may be realized by dedicated hardware. In addition, some or all of the components may be realized by general-purpose or dedicated circuits, processors, etc., or a combination of these. These may be configured by a single chip, or may be configured by multiple chips connected via a bus. Some or all of the components of each device may be realized by a combination of the above-mentioned circuits, etc., and programs. In addition, as a processor, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (field-programmable gate array), a quantum processor (quantum computer control chip), etc. can be used.

In addition, when some or all of the components of the information processing device 100 are realized by multiple information processing devices, circuits, etc., the multiple information processing devices, circuits, etc. may be centrally arranged or distributed. For example, the information processing devices, circuits, etc. may be realized as a client-server system, cloud computing system, etc., in which each is connected via a communication network. In addition, the functions of the information processing device 100 may be provided in the form of SaaS (Software as a Service).

Furthermore, the above-mentioned customer device M0 may have functions similar to those of the above-mentioned information processing device 100.

Next, an information processing method using the information processing device of embodiment 1 will be described. FIG. 7 is a flow chart illustrating an information processing method of embodiment 1. As shown in FIG. 7, the information processing method includes an image data acquisition step (step S11), a countermeasure acquisition step (step S12), an image data conversion step (step S13), an image data determination step (step S14), and a countermeasure notification step (step S15).

First, as shown in step S11, in an image data acquisition step, the image data acquisition unit 110 is caused to acquire representative image data of a failure. Next, as shown in step S12, in a countermeasure acquisition step, the countermeasure acquisition unit 120 is caused to acquire countermeasures in the event of a failure. Note that the order of step S12 is not limited to immediately after step S11, but may be before step S11 or between steps S11 and S15, so long as it is before the countermeasure notification step of step S15.

Next, as shown in step S13, in an image data conversion step, the image data conversion unit 130 converts a part of the time series data 31 of the sensor value acquired by the sensor device SD of the customer device M0 that is the target of the failure judgment into target image data. In the image data conversion step, as described above, for example, the data may be converted into target image data including at least the first target image data and the second target image data.

Next, as shown in step S14, in an image data determination step, the image data determination unit 140 determines whether or not the customer device M0 is faulty by comparing the fault representative image data with the target image data. If the image data determination unit 140 determines that there is a fault (YES) in step S14, then in a countermeasure notification step, the countermeasure notification unit 150 notifies the customer device M0 of a countermeasure (NO) in step S14. On the other hand, if the image data determination unit 140 does not determine that there is a fault (NO) in step S14, the processing ends. In that case, the information processing method may be performed on the next sensor value time series data 31.

Next, the effects of this embodiment will be described. The information processing device 100 of this embodiment determines a failure by comparing target image data with a changed time width of the window frame and representative failure image data. This makes it possible to detect failures and their causes that could not be detected with a fixed window frame. For example, it is possible to detect features 31a and 31b in the time series data 31. This makes it possible to determine the failure of the customer device M0 with high accuracy.

In addition, the image data conversion unit 130 of the information processing device 100 converts the time series data 31 of the customer device M0 connected via the network NW into target image data. This allows various information to be collected online and faults to be determined in real time.

In addition, in the method using machine learning related to failure prediction, it is difficult to organize collected data by type such as temperature and voltage in real time. Therefore, this method notifies the customer of the judgment of whether or not a failure will occur, but it is difficult to automatically suggest how to deal with it specifically, or for the operation side to utilize it in future operations. Therefore, in order to respond by type, it is necessary to communicate each time by email, etc. In addition, even if the content has been dealt with in the past, it may be necessary to check it again, so it lacks accuracy and immediacy. In this way, in the analysis using machine learning, since the data is integrated and characterized, it is difficult to associate what kind of failure there is and how it was dealt with. Therefore, it is not possible to judge the failure with high accuracy and deal with it appropriately. In contrast, the present embodiment collects various information online and can judge the failure in real time, so it is possible to deal with the failure quickly.

When a fault occurs in the customer device M0, in order to respond to the type of sensor value, it was necessary to communicate by e-mail or the like each time, but in this embodiment, it is possible to eliminate the need for e-mail or other communication with the customer device M0, thereby reducing the burden. Also, even if the content of a response was handled in the past, it was sometimes necessary to check it again because it was not properly managed, but in this embodiment, it is stored in the countermeasure storage device 20, so it is possible to manage countermeasures for the failure.

In addition, a fault can be determined for each sensor value acquired by the sensor device SD. Therefore, faults that have occurred in the past can be managed according to their causes, and based on that information, the fault can be determined and a countermeasure can be notified to the customer device M0.

(Embodiment 2)
Next, an information processing system and an information processing device according to the second embodiment will be described. Fig. 8 is a configuration diagram illustrating an information processing system according to the second embodiment. As shown in Fig. 8, the information processing system 2 includes customer devices M01, M11, an information processing device 200, a failure representative image data storage device 10, a countermeasure storage device 20, a customer log data storage device 30, a failure time storage device 40, a reproduction log data storage device 50, a customer image data storage device 60, a failure representative candidate image data storage device 70, and a learning vector data storage device 80. The customer devices M01, M11, and the information processing device 200 function as information processing means. The failure representative image data storage device 10, the countermeasure storage device 20, the customer log data storage device 30, the failure time storage device 40, the reproduction log data storage device 50, the customer image data storage device 60, the failure representative candidate image data storage device 70, and the learning vector data storage device 80 function as a failure representative image data storage means, a countermeasure storage means, a customer log data storage means, a failure time storage means, a reproduction log data storage means, a customer image data storage means, a failure representative candidate image data storage means, and a learning vector data storage means, respectively. In order to avoid cluttering the diagram, the wording of the storage device in each of the storage devices 10 to 80 has been omitted from the diagram.

The information processing system 2 may include customer devices M01, M02, M11, and M12. As in the above-described embodiment, customer devices M01 and M02 are referred to as customer device M0. Customer device M0 is a customer device to be uploaded. Customer devices M11 and M12 are referred to as customer device M1. Customer device M1 is a customer device not to be uploaded. In the figure, two customer devices M0 and two customer devices M1 are shown, but the number of customer devices M0 and M1 may be one or three or more. Customer devices M0 and M1 are connected to the information processing device 200 side via the network NW in a communicable state.

Each of the customer devices M0 and M1 is equipped with various sensor devices SD (not shown). As described above, the various sensor devices SD monitor the temperature, humidity, voltage, vibration, etc. as the status of each part and each component of the installed customer devices M0 and M1.

The customer device M0 that is the upload target uploads various information such as temperature, humidity, voltage, and vibration acquired by the sensor device SD to the information processing device 200 side via the network NW. Therefore, the information processing device 200 acquires various information acquired by the sensor device SD of the customer device M0 via the network NW. On the other hand, the customer device M1 that is not the upload target does not upload at least a part or all of the various information such as temperature, humidity, voltage, and vibration acquired by the sensor device SD to the information processing device 200 side via the network NW. Therefore, the information processing device 200 cannot acquire at least a part or all of the various information acquired by the sensor device SD of the customer device M1 via the network NW. The customer device M1 that is not the upload target is, for example, a local server. The information processing device 200 can acquire the authorized information of the customer device M1 via the network NW. In the following, unless otherwise specified, the authorized information of the customer device M1 will be described as being equivalent to the various information of the customer device M0.

The customer log data storage device 30 is connected to the customer devices M0 and M1 via the network NW in a state in which it can communicate with them. The customer log data storage device 30 is also connected to the information processing device 200 in a state in which it can communicate with them. The customer log data storage device 30 stores various information of the customer device M0. For example, the customer log data storage device 30 stores time series data 31 of various information acquired from the customer device M0. The customer log data storage device 30 outputs the time series data 31 of various information acquired from the customer device M0 to the information processing device 200. Specifically, the customer log data storage device 30 outputs the time series data 31 of the sensor value acquired by the sensor device SD of the customer device M0 that is the subject of the failure judgment to the information processing device 200.

The failure time storage device 40 is connected to the customer log data storage device 30 in a state in which it can communicate with the information processing device 200. The failure time storage device 40 is also connected to the information processing device 200 in a state in which it can communicate with the information processing device 200. The failure time storage device 40 stores the time when the customer device M0 failed. For example, the failure time storage device 40 may obtain the time when the failure occurred from the customer device M0 or from the information processing device 200.

The reproduction log data storage device 50 is connected in a communicable state to the information processing device 200. The reproduction log data storage device 50 stores time series data 31 generated by various sensor devices SD in order to generate learning image data. The reproduction log data storage device 50 stores a sufficient amount of time series data 31 for learning.

The customer image data storage device 60 is connected to the information processing device 200 in a state in which it can communicate with the information processing device 200. The customer image data storage device 60 stores target image data. The target image data is a portion of the time series data of the sensor value acquired by the sensor device SD of the customer device M0 that is the subject of a fault judgment, which has been converted into image data. For example, the target image data visualized by the image data conversion unit 130 is stored.

The failure representative candidate image data storage device 70 is connected in a communicative state to the information processing device 200. The failure representative candidate image data storage device 70 may also be connected in a communicative state to the failure representative image data storage device 10. The failure representative candidate image data storage device 70 stores multiple failure representative candidate image data. The failure representative candidate image data storage device 70 may include, for example, first failure representative candidate image data and second failure representative candidate image data.

The first failure representative candidate image data is an image of a window portion of the past time series data 31 separated by a time width of the first time including the time of the first failure and immediately before the first failure. The first failure representative candidate image data is extracted as the first failure representative image data when a predetermined condition is met. The predetermined condition is, for example, a case where the first failure representative candidate image data is a first candidate image data obtained by imaging a window portion separated by the same time width of the first time for the first failure representative candidate image data, and there are a predetermined first number or more of first candidate image data having a predetermined first similarity or higher.

The second failure representative candidate image data is an image of a window portion of the past time series data 31 separated by a time width of the second time including the time of the second failure and immediately before the second failure. The second failure representative candidate image data is extracted as the second failure representative image data when a predetermined condition is met. The predetermined condition is, for example, when there is a predetermined second number or more of second candidate image data that is an image of a window portion separated by the same time width of the second time for the second failure representative candidate image data and has a predetermined first similarity or higher.

The learning vector data storage device 80 is connected to the information processing device 200 in a communicable state. The learning vector data storage device 80 stores learning vector data. The learning vector data is, for example, vector data used when the image data vectorization unit 160 learns to compress image data into a low-dimensional vector and then restores it, as described below.

Next, the information processing device 200 of the present embodiment will be described. FIG. 9 is a block diagram illustrating an information processing device 200 according to the second embodiment. FIG. 10 is a diagram illustrating the relationship between each configuration of the information processing system 2 according to the second embodiment and each configuration of the information processing device 200. FIG. 10 shows the main relationship between each configuration of the information processing system 2 and each configuration of the information processing device 200, and some relationships between the two are omitted so as not to make the figure too complicated. As shown in FIGS. 9 and 10, the information processing device 200 of the present embodiment further includes an image data vectorization unit 160, an image data extraction unit 170, and a countermeasure input unit 180 compared to the information processing device 100. The image data vectorization unit 160, the image data extraction unit 170, and the countermeasure input unit 180 have the functions of image data vectorization means, image data extraction means, and countermeasure input means, respectively.

The image data conversion unit 130 acquires time series data 31 of the sensor values acquired by the sensor device SD of the customer device M0 that is the target of the fault judgment from the customer log data storage device 30. The image data conversion unit 130 then converts a portion of the acquired time series data 31 into target image data. The image data conversion unit 130 outputs the converted target image data to the customer image data storage device 60.

In addition, the image data conversion unit 130 may generate failure representative candidate image data from customer log data or reproduction log data based on information at the time of failure acquired from the failure time storage device 40. The image data conversion unit 130 outputs the generated failure representative candidate image data to the failure representative candidate image data storage device 70.

In addition, the image data conversion unit 130 acquires time series data 31 of the sensor values acquired by the sensor device SD of the customer device M0 from the reproduction log data storage device 50 in order to generate learning image data. Then, the image data conversion unit 130 converts a part of the time series data 31 into target image data. The image data conversion unit 130 outputs the converted target image data to the image data vectorization unit 160.

The image data determination unit 140 determines whether or not there is a failure in the customer device M0 by comparing the failure representative image data with the target image data, similar to the image data determination unit 140 of embodiment 1. The image data determination unit 140 may determine whether or not there is a failure in the customer device M0 in real time using the target image data converted by the image data conversion unit 130, or may determine whether or not there is a failure in the customer device M0 using the target image data stored in the customer image data storage device 60.

When comparing the failure representative image data with the target image data, the image data determination unit 140 may determine that a failure has occurred in the customer device M0 if the similarity between the two is equal to or greater than a predetermined level. The image data determination unit 140 may also determine that no failure has occurred in the customer device M0 if the similarity between the two is less than a predetermined level. The similarity between the failure representative image data and the target image data may be calculated, for example, from the vector distance between the vector data obtained by vectorizing the failure representative image data and the vector data obtained by vectorizing the target image data. For example, the vector distance between the vector data vectorized by the image data vectorization unit 160 may be used.

Figure 11 is a flow chart illustrating the processing methods of the image data determination unit 140, countermeasure notification unit 150, and image data vectorization unit 160 in the information processing device 200 of embodiment 2. As shown in step S21 of Figure 11, the image data is vectorized. Specifically, the image data vectorization unit 160 vectorizes the failure representative image data and the target image data.

Next, as shown in step S22, the image data determination unit 140 determines whether the vector distance between the vectorized failure representative image data and the vectorized target image data is smaller than a predetermined threshold. In step S22, if the vector distance is smaller than the predetermined threshold (YES), a failure has occurred in the customer device M0. Therefore, as shown in step S23, the countermeasure notification unit 150 notifies the customer device M0 that a failure has occurred and the countermeasure. On the other hand, in step S22, if the vector distance is equal to or greater than the predetermined threshold (NO), the process ends. In this way, the image data determination unit 140 may determine a failure of the customer device M0 by comparing the failure representative image data with the target image data using the vector distance between the failure representative image data and the target image data. Note that the similarity between the failure representative image data and the target image data may be calculated using other calculation methods, not limited to calculation from the vector distance.

The image data vectorization unit 160 is, for example, an autoencoder. The image data vectorization unit 160 vectorizes image data and converts it into vector data. Specifically, for example, the image data vectorization unit 160 vectorizes the failure representative image data and the target image data into vector data. The vectorized data is called vector data. The image data vectorization unit 160 may learn to compress image data into a low-dimensional vector and then restore it. This allows the image data vectorization unit 160 to quickly vectorize the image data into vector data. Therefore, the similarity between the failure representative candidate image data and the candidate image data can be quickly calculated. The image data vectorization unit 160 outputs the vector data used for learning to the learning vector data storage device 80.

The image data extraction unit 170 extracts the failure representative image data. FIG. 12 is a diagram illustrating the failure representative image data extracted by the image data extraction unit 170 in the information processing device 200 according to the second embodiment. As shown in FIG. 12, the image data extraction unit 170 first selects one failure representative candidate image data H11 from the failure representative candidate image data storage device 70. The selected failure representative candidate image data is called the first failure representative candidate image data H11. For example, the image data extraction unit 170 selects the first failure representative candidate image data H11 in the time series data 31 of the element A of the customer device M01. The first failure representative candidate image data H11 is an image of a window frame portion of the time series data 31, which is divided by a time width of the time TT11 including the time T1 and the time immediately before the time T1, by the image data conversion unit 130.

The image data conversion unit 130 generates multiple first candidate image data by imaging a window frame separated by a time width of time TT11 from the time series data 31 of element A of customer devices M01 to M03. For example, the image data conversion unit 130 generates multiple first candidate image data by tracing the window frame back to a predetermined time. The multiple first candidate image data are referred to as first candidate image data H12, H13, H14, ....

The image data vectorization unit 160 vectorizes the generated first candidate image data (H12, H13, H14, ...) and the first failure representative candidate image data H11.

The image data determination unit 140 determines the similarity based on the vector distance between the first candidate image data (H12, H13, H14, ...) and the first failure representative candidate image data H11. The image data determination unit 140 then determines that the similarity between the three first candidate image data H12, H13, and H14 is equal to or greater than a predetermined similarity. In that case, the image data extraction unit 170 counts the number of first candidate image data H12, H13, and H14 that are similar to the failure representative candidate image data H11 as three.

The image data extraction unit 170 extracts the first failure representative candidate image data H11 as the first failure representative image data when there is a predetermined number or more of first candidate image data similar to the first failure representative candidate image data at a predetermined similarity or more. For example, as shown in FIG. 12, the image data extraction unit 170 extracts the first failure representative candidate image data H11 as the first failure representative image data when there are three or more first candidate image data. In this way, the image data extraction unit 170 extracts the first failure representative image data when there are a predetermined first number or more of first candidate image data similar to the first failure representative candidate image data, which is an image of a window frame portion separated by a time width of a first time for the first failure representative candidate image data.

The image data extraction unit 170 may also extract the second failure representative image data by changing the time width from time TT11 to time TT21, which is longer than time TT11. FIG. 13 is a diagram illustrating failure representative image data extracted by the image data extraction unit 170 with a different time width in the information processing device 200 according to the second embodiment. As shown in FIG. 13, the image data extraction unit 170 selects one failure representative candidate image data H21, the time width of which is changed from time TT11 to time TT21, from the failure representative candidate image data storage device 70. The selected failure representative candidate image data is called the second failure representative candidate image data H21. The second failure representative candidate image data H21 is an image of a window frame portion of the time series data 31, which is divided by the time width of time TT21 including the time T1 and immediately before the time T1, by the image data conversion unit 130.

Then, similar to the above-mentioned first failure representative candidate image data, the image data conversion unit 130 generates a plurality of second candidate image data by imaging a window frame separated by the time width of the time TT21 in the time series data 31. For example, the image data conversion unit 130 generates a plurality of second candidate image data by tracing the window frame back to a predetermined time.

The image data vectorization unit 160 vectorizes the generated second candidate image data and the second failure representative candidate image data H21.

The image data determination unit 140 determines the similarity based on the vector distance between the second candidate image data and the second fault representative candidate image data H21.

The image data extraction unit 170 extracts the second failure representative candidate image data H21 as the second failure representative image data when there is a predetermined number or more of second candidate image data having a predetermined similarity or higher in the second failure representative candidate image data H21. In this way, the image data extraction unit 170 extracts the second failure representative image data when there is a predetermined second number or more of second candidate image data having a predetermined second similarity or higher, which is second candidate image data obtained by imaging a window frame portion separated by a time width of the second time for the second failure representative candidate image data. In this way, the image data extraction unit 170 extracts failure representative image data including the first failure representative image data and the second failure representative image data. By extracting failure representative image data having different time widths, the feature 31c of the time series data 31 can be detected.

The above-mentioned method for extracting the representative failure image data will be described using a flowchart. FIG. 14 is a flowchart illustrating a method for extracting the representative failure image data in the information processing device 200 according to the second embodiment. As shown in step S31 of FIG. 14, the image data conversion unit 130 acquires time series data 31 from the customer log data storage device 30 of the customer device M0. Next, as shown in step S32, the image data conversion unit 130 sets the time width of the window frame. Next, as shown in step S33, it is determined whether the window frame is within a range up to a predetermined time. In step S33, if the answer is YES that the window frame is within the range up to the predetermined time, the window frame is set at the position going back as shown in step S34. This sets the candidate image data.

Next, as shown in step S35, the image data vectorization unit 160 vectorizes the window frame portion for the two image data to be compared, the failure representative candidate image data and the candidate image data. Next, as shown in step S36, the vector distance is compared to determine whether it is smaller than a predetermined threshold. If the vector distance is smaller than the predetermined threshold (YES) in step S36, the candidate image data is stored in the failure representative candidate image data storage device 70 as shown in step S37. Then, as shown in step S38, it is determined whether there are other elements within the same period and in the same window frame. On the other hand, if the vector distance is equal to or greater than the predetermined threshold (NO) in step S36, it is determined whether there are other elements within the same period and in the same window frame as shown in step S38.

If the answer in step S38 is YES, meaning that there are other elements in the same period and in the same window frame, then the next element (for example, voltage after temperature) is set as shown in step S39. Then, steps S36 to S38 are repeated. On the other hand, if the answer in step S38 is NO, meaning that there are no other elements in the same period and in the same window frame, then the process proceeds to step S33, where it is determined whether the window frame is within a predetermined time range.

If the answer to step S33 is NO, meaning that the window frame is outside the range up to the predetermined time, then as shown in step S40, the time width of the window frame is expanded by a predetermined size at the first position in the time series data 31. Then, as shown in step S41, it is determined whether the expanded window frame is equal to or larger than a predetermined maximum window frame size. If the answer to step S41 is NO, meaning that the expanded window frame is smaller than the predetermined maximum window frame size, then the process returns to step S33. On the other hand, if the answer to step S41 is YES, meaning that the expanded window frame is equal to or larger than the predetermined maximum window frame size, then the process ends.

In step S36, if the number of candidate image data judged to be similar to the failure representative candidate image data is equal to or greater than a predetermined number, the image data extraction unit 170 extracts the failure representative candidate image data as the failure representative image data.

The countermeasure input unit 180 inputs countermeasures to the countermeasure storage device 20. The countermeasure input unit 180 inputs countermeasures used at the time of past failures to the countermeasure storage device 20. For example, the countermeasure input unit 180 inputs a first countermeasure to the countermeasure storage device 20 when a failure was determined in the time series data 31 of the first sensor value in the past. The countermeasure input unit 180 also inputs a second countermeasure to the countermeasure storage device 20 when a failure was determined in the time series data 31 of the second sensor value in the past. Furthermore, the countermeasure input unit 180 inputs a third countermeasure to the countermeasure storage device 20 when a failure was determined in the time series data 31 of the first sensor value and the second sensor value in the past. The third countermeasure may include a countermeasure different from the first countermeasure and the second countermeasure. The countermeasure input unit 180 may also input past countermeasures for each similar pattern of the representative failure image data.

Next, the effects of this embodiment will be described. In the information processing device 200 of this embodiment, the image data extraction unit 170 counts candidate image data similar to the failure representative candidate image data. Then, if there is a certain amount of similar candidate image data, the image data extraction unit 170 extracts the failure representative candidate image data as the failure representative image data. Therefore, it is possible to extract failure representative image data that has a cause, rather than a sudden change in the sensor value. Therefore, it is possible to determine the occurrence of a failure with high accuracy.

In addition, the failure representative image data includes, in addition to the first failure representative image data, first failure representative image data with a time width greater than the time width of the first failure representative image data. Therefore, it is possible to detect features 31c of the time series data 31 that could not be detected in a short window. This makes it possible to determine the failure of the customer device M0 with high accuracy. Other configurations and effects are included in the description of embodiment 1.

(Embodiment 3)
Next, an information processing system and an information processing device according to a third embodiment will be described. In this embodiment, a failure is determined using pseudo-fault image data. FIG. 15 is a configuration diagram illustrating an information processing system according to the third embodiment. As shown in FIG. 15, the information processing system 3 further includes a pseudo-fault image data storage device 90. The pseudo-fault image data storage device 90 functions as a pseudo-fault image data storage means. The pseudo-fault image data storage device 90 is connected to the information processing device 300 in a communicable state. The pseudo-fault image data storage device 90 stores pseudo-fault image data.

The pseudo-fault image data is an image of a portion of the time series data of pseudo sensor values formed to show a pseudo-fault state of the customer device M0. For example, by using pseudo-fault image data manually created from data empirically believed to cause failures in the customer device M0, it is possible to check whether there are any similar failures in the past. In addition, in order to investigate the cause of a failure in the customer device M0, the relationship between each component and the failure can be investigated by forming pseudo sensor values in which the sensor values of specific components, etc. are pseudo-changed as parameters.

Figure 16 is a block diagram illustrating an information processing device according to embodiment 3. As shown in Figure 16, the information processing device 300 further includes a pseudo-fault image data acquisition unit 190. The pseudo-fault image data acquisition unit 190 has a function as a pseudo-fault image data acquisition means. The pseudo-fault image data acquisition unit 190 acquires pseudo-fault image data stored in the pseudo-fault image data storage device 90. In this embodiment, the image data determination unit 140 determines a failure in the pseudo-fault state by comparing the pseudo-fault image data with the failure representative image data.

Figure 17 is a flow chart illustrating a method of processing pseudo-fault image data in the information processing device 300 of embodiment 3. As shown in step S51 of Figure 17, failure representative image data and pseudo-fault image data are acquired as image data. Specifically, the image data acquisition unit 110 acquires failure representative image data from the failure representative image data storage device 10. In addition, the pseudo-fault image data acquisition unit 190 acquires pseudo-fault image data from the pseudo-fault image data storage device 90.

Next, as shown in step S52, the image data vectorization unit 160 vectorizes the failure representative image data and the pseudo-failure image data. Next, as shown in step S53, the image data determination unit 140 determines whether the vector distance between the vectorized failure representative image data and the pseudo-failure image data is smaller than a predetermined threshold. In step S53, if the vector distance is smaller than the predetermined threshold (YES), the countermeasure notification unit 150 notifies the customer device M0 that the pseudo-failure state corresponds to a failure and the countermeasure (NO), as shown in step S54. On the other hand, in step S53, if the vector distance is equal to or greater than the predetermined threshold (NO), the countermeasure notification unit 150 notifies the customer device M0 that the pseudo-failure state does not correspond to a failure and the countermeasure (NO), as shown in step S54. Then, the process ends.

Next, the effects of this embodiment will be described. In the information processing device 300 of this embodiment, the image data determination unit 140 determines a failure in the pseudo-failure state by comparing pseudo-failure image data formed to show the pseudo-failure state of the customer device M0 with the representative failure image data. This makes it possible to confirm where and for what reason the failure occurred in the customer device M0. This can be utilized in future failure countermeasures. Other configurations and effects are included in the description of embodiments 1 and 2.

(Embodiment 4)
Next, an information processing system and an information processing device according to a fourth embodiment will be described. In this embodiment, the customer device M0 that is the upload target and the customer device M1 that is not the upload target will be described in detail. First, the customer device M0 that is the upload target will be described.

Figure 18 is a block diagram illustrating an example of a customer device M0 to be uploaded in an information processing system relating to embodiment 4. As shown in Figure 18, in the information processing system 4, the customer device M0 is connected to the information processing device 400 via a network NW. The customer device M0 is equipped with one or more sensor devices SD, a display unit DP, and a log data storage unit LD. The sensor device SD, the display unit DP, and the log data storage unit LD each have the functions of a sensing means, a display means, and a log data storage means. The information processing device 400 is similar to the information processing devices 100 to 300.

The sensor device SD acquires the sensor value of the customer device M0. The sensor device SD outputs the acquired sensor value to the log data storage unit LD as various information. The sensor device SD may also output the acquired sensor value as various information to the information processing device 400 via the network NW. The display unit DP displays the countermeasure notified by the countermeasure notification unit 150 of the information processing device 400. The display unit DP displays the countermeasure when a failure is determined. The log data storage unit LD stores log data of the customer device M0. The log data includes various information such as the sensor value acquired by the sensor device SD. The log data stored in the log data storage unit LD may be output to the information processing device 400 via the network NW.

Next, the non-uploaded customer device M1 will be described. FIG. 19 is a block diagram illustrating a non-uploaded customer device M1 in an information processing system according to embodiment 4. As shown in FIG. 19, in the information processing system 4, the customer device M1 is connected to the information processing device 400 via a network NW. The customer device M1 includes one or more sensor devices SD, a display unit DP, and a log data storage unit LD, as well as an information processing unit 400a, a failure representative image data storage unit 10a, and a countermeasure storage unit 20a. The information processing unit 400a functions as an information processing means. The failure representative image data storage unit 10a and the countermeasure storage unit 20a function as a failure representative image data storage means and a countermeasure storage means, respectively.

The failure representative image data storage unit 10a has the same functions as the failure representative image data storage device 10. Specifically, the failure representative image data storage unit 10a stores failure representative image data that is an image of a part of the time series data 31 of the sensor value acquired by the sensor device of the customer device M0 when the customer device M0 fails. The failure representative image data storage unit 10a may be, for example, a copy of the failure representative image data storage device 10. The failure representative image data storage unit 10a may store failure representative image data of the customer device M1. The countermeasure storage unit 20a may store countermeasures when the customer device M0 fails, or may store countermeasures when the customer device M1 fails.

The information processing unit 400a has an image data acquisition unit 110, a countermeasure acquisition unit 120, an image data conversion unit 130, and an image data determination unit 140 similar to those installed in the information processing device 400. The image data acquisition unit 110 acquires failure representative image data from the failure representative image data storage unit 10a. The countermeasure acquisition unit 120 acquires countermeasures at the time of failure from the countermeasure storage unit 20a. The image data conversion unit 130 converts a part of the time series data 31 of the customer device M1 stored in the log data storage unit LD into target image data. The image data conversion unit 130 converts into target image data including the first target image data and the second target image data. The image data determination unit 140 judges the failure of the customer device M1 by comparing the failure representative image data with the target image data. The display unit DP displays the countermeasure acquired by the countermeasure acquisition unit 120 from the countermeasure storage unit 20a when the customer device M1 is judged to be broken.

Next, the effects of this embodiment will be described. In this embodiment, the customer device M0 is connected to the information processing device 400, and various information about the customer device M0 is output to the information processing device 400. When the information processing device 400 determines that the customer device M0 has failed, the information processing device 400 notifies the customer device M0 of measures to be taken in the event of the failure. This allows the failure to be dealt with promptly.

In addition, in cases where it is not desired to upload various information to the information processing device 400, as in the case of customer device M1, the device is provided with a failure representative image data storage unit 10a having the same functions as the failure representative image data storage unit 10. Furthermore, customer device M1 is provided with an information processing unit 400a having the same functions as the information processing device 400. This allows customer device M1 to obtain failure judgment results from the built-in failure representative image data storage unit 10a and information processing unit 400a. Other configurations and effects are included in the description of embodiments 1 to 3.

(Embodiment 5)
Next, a fifth embodiment will be described. This embodiment is an example of a search method that considers combinations of elements. The search for combinations is performed after the search for the representative failure candidate image data for each element is completed.

First, combinations are searched for in the failure representative candidate image data, starting from element 1. Combinations of up to n elements are considered. n is an arbitrary number and can be set in advance. For example, if the number of elements is five, they are arranged as element 1, element 2, element 3, element 4, element 5. Here, element 1, element 2, ... are assumed to be, for example, temperature, humidity, ... For example, if the number of elements to be considered for combination is two, two combinations are considered from the five elements. Therefore, in searching in the order of element 1, element 2, element 3, element 4, element 5, when the elements to be considered for combination are set to "1" and the elements not to be considered are set to "0", the following 10 combinations are possible.

11000, 10100, 10010,...00011

This is treated as a binary number, and converted to a decimal number by inverting the numbers so that the numbers are arranged in ascending order, for example, 00110 → 01100. For example, "10100" is converted to a decimal number as "00101", and stored as "5" in the "Designation" column of the following management data table.

The column for each element in the management data table stores the number of hits (the number of similar candidate image data) during the search. Combinations are made starting with the smallest element. Numbers are stored in ascending order starting from 0 in the "item" of the management data table, and candidate image data similar to the representative failure candidate image data is stored in the "reference image" column in accordance with the "designation" column. In addition, in order to take combinations into account, data other than image data, such as maximum temperature values, may also be handled.

Note that the present disclosure is not limited to the above-described embodiments, and can be modified as appropriate without departing from the spirit of the present disclosure. For example, it is also possible to combine the configurations of embodiments 1 to 5.

Also within the scope of the technical idea of the embodiment is an information processing program that causes a computer to read and execute the above-mentioned information processing method. The information processing program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The information processing program may be transmitted on a temporary computer-readable medium or a communication medium. By way of example and not limitation, a temporary computer-readable medium or a communication medium includes electrical, optical, acoustic, or other forms of propagating signals.

Some or all of the above embodiments may also be described as, but are not limited to, the following notes:

(Appendix 1)
an image data acquisition means for acquiring failure representative image data obtained by imaging a part of time series data of sensor values acquired by a sensor device of the customer device when the customer device fails;
A countermeasure acquisition means for acquiring a countermeasure at the time of the failure;
an image data conversion means for converting a part of the time series data of the sensor value acquired by the sensor device of the customer device that is a target for failure determination into target image data;
an image data determination means for determining a failure of the customer device by comparing the representative failure image data with the target image data;
a countermeasure notification means for notifying the customer device of the countermeasure when the customer device is determined to be faulty;
Equipped with
The image data conversion means
a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data into a first time span, into an image while moving the first window frame back along the time axis of the time series data to a predetermined time at a first back time interval;
a plurality of second target image data pieces obtained by converting each second window frame into an image while moving the second window frame, which divides the time series data into a second time period greater than the first time period, back along the time axis of the time series data to the predetermined time at a predetermined second back time interval;
converting the target image data into the target image data including
Information processing device.
(Appendix 2)
Further comprising an image data extraction unit for extracting the failure representative image data including the first failure representative image data and the second failure representative image data,
The image data extraction means includes:
extracting, from the time-series data, first failure representative candidate image data obtained by imaging a window portion divided by the time width of the first time including the time of the failure and immediately before the failure, the first failure representative candidate image data being first candidate image data obtained by imaging the window portion divided by the time width of the first time, when there are a first number or more pieces of first candidate image data having a first similarity or higher;
extracting, from the time-series data, second failure representative candidate image data obtained by imaging a window portion divided by the time width of the second time including the time of the failure and immediately before the failure, the second candidate image data obtained by imaging the window portion divided by the time width of the second time, when there are a second number or more of the second candidate image data having a second similarity or higher; the second failure representative candidate image data is extracted as the second failure representative image data;
2. The information processing device according to claim 1.
(Appendix 3)
The failure representative image data and the target image data are vectorized into vector data by an image data vectorization unit.
the image data determination means determines whether the customer device is faulty by comparing the representative fault image data with the target image data using a vector distance between the representative fault image data and the target image data;
3. The information processing device according to claim 1 or 2.
(Appendix 4)
The sensor value is
a first sensor value acquired by a first sensor device;
a second sensor value acquired by a second sensor device different from the first sensor device;
Including,
The image data determination means
comparing the target image data obtained by imaging a part of the time-series data of the first sensor value with the failure representative image data obtained by imaging a part of the time-series data of the first sensor value;
comparing the target image data obtained by imaging a part of the time-series data of the second sensor value with the failure representative image data obtained by imaging a part of the time-series data of the second sensor value;
The countermeasure acquisition means includes:
acquiring a first countermeasure in the case where a fault is determined in the time-series data of the first sensor value;
acquiring a second countermeasure in the case where a fault is determined in the time-series data of the second sensor value;
acquiring a third countermeasure in the case where a fault is determined in the time-series data of the first sensor value and the second sensor value;
4. The information processing device according to claim 1 .
(Appendix 5)
a pseudo-fault image data acquisition means for acquiring pseudo-fault image data obtained by imaging a part of the time-series data of the pseudo sensor value formed so as to indicate a pseudo-fault state of the customer device;
The image data determination means
determining a fault in the pseudo-fault state by comparing the pseudo-fault image data with the fault representative image data;
5. The information processing device according to claim 1 .
(Appendix 6)
a customer device including a sensor device for acquiring a sensor value;
an information processing device connected to the customer device via a network;
a failure representative image data storage device that stores failure representative image data obtained by imaging a part of the time series data of the sensor value acquired by the sensor device of the customer device when the customer device fails;
a countermeasure storage device that stores countermeasures to be taken when the failure occurs;
Equipped with
The information processing device includes:
A failure representative image data acquisition means for acquiring the failure representative image data;
A countermeasure acquisition means for acquiring a countermeasure at the time of the failure;
an image data conversion means for converting a part of the time series data of the sensor value acquired by the sensor device of the customer device that is a target for failure determination into target image data;
an image data determination means for determining a failure of the customer device by comparing the representative failure image data with the target image data;
a countermeasure notification means for notifying the customer device of the countermeasure when the customer device is determined to be faulty;
having
The image data conversion means
a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data into a first time span, into an image while moving the first window frame back along the time axis of the time series data to a predetermined time at a first back time interval;
a plurality of second target image data pieces obtained by converting each second window frame into an image while moving the second window frame, which divides the time series data into a second time period greater than the first time period, back along the time axis of the time series data to the predetermined time at a predetermined second back time interval;
converting the target image data into
Information processing system.
(Appendix 7)
The information processing device further includes an image data extraction unit for extracting the failure representative image data including first failure representative image data and second failure representative image data,
The image data extraction means includes:
extracting, from the time-series data, first failure representative candidate image data obtained by imaging a window portion divided by the time width of the first time including the time of the failure and immediately before the failure, the first failure representative candidate image data being first candidate image data obtained by imaging the window portion divided by the time width of the first time, when there are a first number or more pieces of first candidate image data having a first similarity or higher;
extracting, from the time-series data, second failure representative candidate image data obtained by imaging a window portion divided by the time width of the second time including the time of the failure and immediately before the failure, the second candidate image data obtained by imaging the window portion divided by the time width of the second time, when there are a second number or more of the second candidate image data having a second similarity or higher; the second failure representative candidate image data is extracted as the second failure representative image data;
7. The information processing system according to claim 6.
(Appendix 8)
The image data vectorization means further includes an image data vectorization means for vectorizing the failure representative image data and the target image data into vector data,
the image data determination means determines whether the customer device is faulty by comparing the representative fault image data with the target image data using a vector distance between the representative fault image data and the target image data;
8. The information processing system according to claim 6 or 7.
(Appendix 9)
The sensor value is
a first sensor value acquired by a first sensor device;
a second sensor value acquired by a second sensor device different from the first sensor device;
Including,
The image data determination means
comparing the target image data obtained by imaging a part of the time-series data of the first sensor value with the failure representative image data obtained by imaging a part of the time-series data of the first sensor value;
comparing the target image data obtained by imaging a part of the time-series data of the second sensor value with the failure representative image data obtained by imaging a part of the time-series data of the second sensor value;
The countermeasure acquisition means includes:
acquiring a first countermeasure in the case where a fault is determined in the time-series data of the first sensor value;
acquiring a second countermeasure in the case where a fault is determined in the time-series data of the second sensor value;
acquiring a third countermeasure in the case where a fault is determined in the time-series data of the first sensor value and the second sensor value;
An information processing system according to any one of appendix 6 to 8.
(Appendix 10)
The information processing device further includes a pseudo-fault image data acquisition means for acquiring pseudo-fault image data obtained by imaging a part of the time-series data of the pseudo sensor value formed so as to indicate a pseudo-fault state of the customer device,
The image data determination means
determining a fault in the pseudo-fault state by comparing the pseudo-fault image data with the fault representative image data;
An information processing system according to any one of appendix 6 to 9.
(Appendix 11)
A first customer device including a sensor device for acquiring a sensor value;
a second customer device including the sensor device for acquiring the sensor value;
an information processing device connected to the first customer device via a network;
Equipped with
The second customer device is
a failure representative image data storage means for storing failure representative image data that is an image of a part of the time series data of the sensor value acquired by the sensor device of the first customer device when the first customer device fails;
A countermeasure storage means for storing countermeasures to be taken when the failure occurs;
An image data acquisition means for acquiring the failure representative image data;
A countermeasure acquisition means for acquiring a countermeasure at the time of the failure;
an image data conversion means for converting a part of the time series data of the sensor value acquired by the sensor device of the second customer apparatus that is a target for failure determination into target image data;
an image data determination means for determining a failure of the second customer device by comparing the representative failure image data with the target image data;
a display means for displaying the countermeasure when the malfunction is determined;
having
The image data conversion means
a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data into a first time span, into an image while moving the first window frame back along the time axis of the time series data to a predetermined time at a first back time interval;
a plurality of second target image data pieces obtained by converting each second window frame into an image while moving the second window frame, which divides the time series data into a second time period greater than the first time period, back along the time axis of the time series data to the predetermined time at a predetermined second back time interval;
converting the target image data into the target image data including
Information processing system.
(Appendix 12)
The second customer device further includes an image data extraction unit for extracting the failure representative image data including the first failure representative image data and the second failure representative image data,
The image data extraction means includes:
extracting, from the time-series data, first failure representative candidate image data obtained by imaging a window portion divided by the time width of the first time including the time of the failure and immediately before the failure, the first failure representative candidate image data being first candidate image data obtained by imaging the window portion divided by the time width of the first time, when there are a first number or more pieces of first candidate image data having a first similarity or higher;
extracting, from the time-series data, second failure representative candidate image data obtained by imaging a window portion divided by the time width of the second time including the time of the failure and immediately before the failure, the second candidate image data obtained by imaging the window portion divided by the time width of the second time, when there are a second number or more of the second candidate image data having a second similarity or higher; the second failure representative candidate image data is extracted as the second failure representative image data;
12. The information processing system according to claim 11.
(Appendix 13)
The image data vectorization means further includes an image data vectorization means for vectorizing the failure representative image data and the target image data into vector data,
the image data determination means determines whether the second customer device has a failure by comparing the representative failure image data with the target image data using a vector distance between the representative failure image data and the target image data;
13. The information processing system according to claim 11 or 12.
(Appendix 14)
The sensor value is
a first sensor value acquired by a first sensor device;
a second sensor value acquired by a second sensor device different from the first sensor device;
Including,
The image data determination means
comparing the target image data obtained by imaging a part of the time-series data of the first sensor value with the failure representative image data obtained by imaging a part of the time-series data of the first sensor value;
comparing the target image data obtained by imaging a part of the time-series data of the second sensor value with the failure representative image data obtained by imaging a part of the time-series data of the second sensor value;
The countermeasure acquisition means includes:
acquiring a first countermeasure in the case where a fault is determined in the time-series data of the first sensor value;
acquiring a second countermeasure in the case where a malfunction is determined in the time-series data of the second sensor value;
acquiring a third countermeasure in the case where a fault is determined in the time-series data of the first sensor value and the second sensor value;
14. An information processing system according to any one of appendixes 11 to 13.
(Appendix 15)
The second customer device further includes a pseudo-fault image data acquisition means for acquiring pseudo-fault image data obtained by imaging a part of the time-series data of the pseudo sensor value formed so as to indicate a pseudo-fault state of the second customer device,
The image data determination means
determining a fault in the pseudo-fault state by comparing the pseudo-fault image data with the fault representative image data;
15. An information processing system according to any one of appendices 11 to 14.
(Appendix 16)
A sensor device that acquires a sensor value;
a display means for displaying a countermeasure to be taken when a malfunction is determined;
Equipped with
an image data acquisition means for acquiring failure representative image data obtained by imaging a part of the time series data of the sensor value acquired by the sensor device of the apparatus when the apparatus fails;
A countermeasure acquisition means for acquiring a countermeasure at the time of the failure;
an image data conversion means for converting a part of the time series data of the sensor values acquired by the sensor device of the apparatus that is the subject of a failure judgment into target image data;
an image data determination means for determining a failure of the device by comparing the representative failure image data with the target image data;
a countermeasure notification means for notifying the device of the countermeasure when the device is determined to be faulty;
via a network to an information processing device having
The image data conversion means
a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data into a first time span, into an image while moving the first window frame back along the time axis of the time series data to a predetermined time at a first back time interval;
a plurality of second target image data pieces obtained by converting each second window frame into an image while moving the second window frame, which divides the time series data into a second time period greater than the first time period, back along the time axis of the time series data to the predetermined time at a predetermined second back time interval;
The information processing device converts the target image data into the target image data including
Customer equipment.
(Appendix 17)
The information processing device further includes an image data extraction unit for extracting the failure representative image data including first failure representative image data and second failure representative image data,
The image data extraction means includes:
extracting, from the time-series data, first failure representative candidate image data obtained by imaging a window portion divided by the time width of the first time including the time of the failure and immediately before the failure, the first failure representative candidate image data being first candidate image data obtained by imaging the window portion divided by the time width of the first time, when there are a first number or more pieces of first candidate image data having a first similarity or higher;
a second failure representative candidate image data obtained by imaging a window portion divided by the time width of the second time including the time of the failure and immediately before the failure out of the time series data, and a second candidate image data obtained by imaging the window portion divided by the time width of the second time, when the second candidate image data having a predetermined second similarity or higher is a predetermined second number or more, the second failure representative candidate image data is extracted as the second failure representative image data.
17. The customer device of claim 16.
(Appendix 18)
The image data vectorization means further includes an image data vectorization means for vectorizing the failure representative image data and the target image data into vector data,
the image data determination means is connected to the information processing device and determines a failure of the customer device by comparing the representative failure image data with the target image data using a vector distance;
18. A customer device as described in claim 16 or 17.
(Appendix 19)
The sensor value is
a first sensor value acquired by a first sensor device;
a second sensor value acquired by a second sensor device different from the first sensor device;
Including,
The image data determination means
comparing the target image data obtained by imaging a part of the time-series data of the first sensor value with the failure representative image data obtained by imaging a part of the time-series data of the first sensor value;
comparing the target image data obtained by imaging a part of the time-series data of the second sensor value with the failure representative image data obtained by imaging a part of the time-series data of the second sensor value;
The countermeasure acquisition means includes:
acquiring a first countermeasure in the case where a fault is determined in the time-series data of the first sensor value;
acquiring a second countermeasure in the case where a malfunction is determined in the time-series data of the second sensor value;
acquiring a third countermeasure in the case where a fault is determined in the time-series data of the first sensor value and the second sensor value;
19. A customer device according to any one of appendix 16 to 18.
(Appendix 20)
The information processing device further includes a pseudo-fault image data acquisition means for acquiring pseudo-fault image data obtained by imaging a part of the time-series data of the pseudo sensor value formed so as to indicate a pseudo-fault state of the customer device,
The image data determination means
a device connected to the information processing device that determines a failure in the pseudo-failure state by comparing the pseudo-failure image data with the failure representative image data;
20. A customer device according to any one of appendix 16 to 19.
(Appendix 21)
A sensor device that acquires a sensor value;
An information processing means;
a display means for displaying a countermeasure to be taken when a malfunction is determined;
Equipped with
The information processing means includes:
an image data acquisition means for acquiring failure representative image data obtained by imaging a part of the time series data of the sensor value acquired by the sensor device of the apparatus when the apparatus fails;
a countermeasure acquisition means for acquiring the countermeasure when the failure occurs;
an image data conversion means for converting a part of the time series data of the sensor values acquired by the sensor device of the apparatus that is the subject of a failure judgment into target image data;
an image data determination means for determining a failure of the device by comparing the representative failure image data with the target image data;
having
The image data conversion means
a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data into a first time span, into an image while moving the first window frame back along the time axis of the time series data to a predetermined time at a first back time interval;
a plurality of second target image data pieces obtained by converting each second window frame into an image while moving the second window frame, which divides the time series data into a second time period greater than the first time period, back along the time axis of the time series data to the predetermined time at a predetermined second back time interval;
converting the target image data into
Customer equipment.
(Appendix 22)
The information processing means further includes an image data extraction means for extracting the failure representative image data including first failure representative image data and second failure representative image data,
The image data extraction means includes:
extracting, from the time-series data, first failure representative candidate image data obtained by imaging a window portion divided by the time width of the first time including the time of the failure and immediately before the failure, the first failure representative candidate image data being first candidate image data obtained by imaging the window portion divided by the time width of the first time, when there are a first number or more pieces of first candidate image data having a first similarity or higher;
extracting, from the time-series data, second failure representative candidate image data obtained by imaging a window portion divided by the time width of the second time including the time of the failure and immediately before the failure, the second candidate image data obtained by imaging the window portion divided by the time width of the second time, when there are a second number or more of the second candidate image data having a second similarity or higher; the second failure representative candidate image data is extracted as the second failure representative image data;
22. The customer device of claim 21.
(Appendix 23)
The image data vectorization means further includes an image data vectorization means for vectorizing the failure representative image data and the target image data into vector data,
the image data determination means determines whether the customer device is faulty by comparing the representative fault image data with the target image data using a vector distance between the representative fault image data and the target image data;
23. A customer device as described in claim 21 or 22.
(Appendix 24)
The sensor value is
a first sensor value acquired by a first sensor device;
a second sensor value acquired by a second sensor device different from the first sensor device;
Including,
The image data determination means
comparing the target image data obtained by imaging a part of the time-series data of the first sensor value with the failure representative image data obtained by imaging a part of the time-series data of the first sensor value;
comparing the target image data obtained by imaging a part of the time-series data of the second sensor value with the failure representative image data obtained by imaging a part of the time-series data of the second sensor value;
The countermeasure acquisition means includes:
acquiring a first countermeasure in the case where a fault is determined in the time-series data of the first sensor value;
acquiring a second countermeasure in the case where a fault is determined in the time-series data of the second sensor value;
acquiring a third countermeasure in the case where a fault is determined in the time-series data of the first sensor value and the second sensor value;
24. A customer device according to any one of appendix 21 to 23.
(Appendix 25)
The information processing means further includes a pseudo-fault image data acquisition means for acquiring pseudo-fault image data obtained by imaging a part of the time-series data of the pseudo sensor value formed so as to indicate a pseudo-fault state of the device,
The image data determination means
determining a fault in the pseudo-fault state by comparing the pseudo-fault image data with the fault representative image data;
25. A customer device according to any one of appendix 21 to 24.
(Appendix 26)
an image data acquisition step of acquiring failure representative image data obtained by imaging a part of time series data of sensor values acquired by a sensor device of the customer device when the customer device is broken;
a countermeasure acquisition step of acquiring a countermeasure for the failure;
an image data conversion step of converting a part of the time series data of the sensor values acquired by the sensor device of the customer device that is a target for failure determination into target image data;
an image data determination step of determining whether or not the customer device has a malfunction by comparing the representative malfunction image data with the target image data;
a countermeasure notification step of notifying the customer device of the countermeasure when the fault is determined;
Equipped with
In the image data conversion step,
a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data into a first time span, into an image while moving the first window frame back along the time axis of the time series data to a predetermined time at a first back time interval;
a plurality of second target image data pieces obtained by converting each second window frame into an image while moving the second window frame, which divides the time series data into a second time greater than the first time, back along the time axis of the time series data to the predetermined time at a predetermined second back time interval;
converting the target image data into the target image data including
Information processing methods.
(Appendix 27)
A failure representative image data extraction step of extracting the failure representative image data including the first failure representative image data and the second failure representative image data,
In the failure representative image data extraction step,
extracting, from the time-series data, first failure representative candidate image data obtained by imaging a window portion divided by the time width of the first time including the time of the failure and immediately before the failure, the first candidate image data obtained by imaging the window portion divided by the time width of the first time, when there are a first number or more pieces of first candidate image data having a first similarity or higher;
extracting, from the time-series data, second failure representative candidate image data obtained by imaging a window portion divided by the time width of the second time including the time of the failure and immediately before the failure, the second candidate image data obtained by imaging the window portion divided by the time width of the second time, when there are a second number or more of the second candidate image data having a second similarity or higher; the second failure representative candidate image data is extracted as the second failure representative image data;
27. The information processing method according to claim 26.
(Appendix 28)
The failure representative image data and the target image data are vectorized into vector data,
In the image data determination step,
a comparison is made between the representative failure image data and the target image data using a vector distance, thereby determining whether or not the customer device has a failure;
28. The information processing method according to claim 26 or 27.
(Appendix 29)
The sensor value is
a first sensor value acquired by a first sensor device;
a second sensor value acquired by a second sensor device different from the first sensor device;
Including,
In the image data determination step,
comparing the target image data obtained by imaging a part of the time-series data of the first sensor value with the failure representative image data obtained by imaging a part of the time-series data of the first sensor value;
comparing the target image data obtained by imaging a part of the time-series data of the second sensor value with the failure representative image data obtained by imaging a part of the time-series data of the second sensor value;
In the countermeasure acquisition step,
acquiring a first countermeasure in the case where a fault is determined in the time-series data of the first sensor value;
acquiring a second countermeasure in the case where a fault is determined in the time-series data of the second sensor value;
acquiring a third countermeasure in the case where a malfunction is determined in the time-series data of the first sensor value and the second sensor value;
29. An information processing method according to any one of appendices 26 to 28.
(Appendix 30)
The method further includes a pseudo-fault image data acquisition step of acquiring pseudo-fault image data by imaging the time-series data of the pseudo sensor values formed so as to indicate a pseudo-fault state of the customer device,
In the image data determination step,
determining a failure of the simulated failure state by comparing the simulated failure image data with the failure representative image data;
30. An information processing method according to any one of appendices 26 to 29.
(Appendix 31)
an image data acquisition step of acquiring failure representative image data obtained by imaging a part of time series data of sensor values acquired by a sensor device of the customer device when the customer device is broken;
a countermeasure acquisition step of acquiring a countermeasure for the failure;
an image data conversion step of converting a part of the time series data of the sensor values acquired by the sensor device of the customer device that is a target for failure determination into target image data;
an image data determination step of determining whether the customer device has a malfunction by comparing the representative malfunction image data with the target image data;
a countermeasure notification step of notifying the customer device of the countermeasure when the fault is determined;
When you run the following on your computer:
In the image data conversion step,
a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data into a first time span, into an image while moving the first window frame back along the time axis of the time series data to a predetermined time at a first back time interval;
a plurality of second target image data pieces obtained by converting each second window frame into an image while moving the second window frame, which divides the time series data into a second time greater than the first time, back along the time axis of the time series data to the predetermined time at a predetermined second back time interval;
converting the target image data into the target image data including
A non-transitory computer-readable medium having stored thereon an information processing program that causes a computer to execute the following:
(Appendix 32)
A failure representative image data extraction step of extracting the failure representative image data including the first failure representative image data and the second failure representative image data is further performed by the computer;
In the failure representative image data extraction step,
extracting, from the time-series data, first failure representative candidate image data obtained by imaging a window portion divided by the time width of the first time including the time of the failure and immediately before the failure, the first candidate image data obtained by imaging the window portion divided by the time width of the first time, when there are a first number or more pieces of first candidate image data having a first similarity or higher;
extracting, from the time-series data, second failure representative candidate image data obtained by imaging a window portion divided by the time width of the second time including the time of the failure and immediately before the failure, the second candidate image data obtained by imaging the window portion divided by the time width of the second time, when there are a second number or more of the second candidate image data having a second similarity or higher; the second failure representative candidate image data is extracted as the second failure representative image data;
A non-transitory computer-readable medium having stored thereon the information processing program described in Appendix 31 that causes a computer to execute the steps described above.
(Appendix 33)
The computer further executes an image data vectorization step of vectorizing the failure representative image data and the target image data into vector data;
In the image data determination step,
a comparison is made between the representative failure image data and the target image data using a vector distance, thereby determining whether or not the customer device has a failure;
A non-transitory computer-readable medium having stored thereon an information processing program according to claim 31 or 32 that causes a computer to execute the above-mentioned steps.
(Appendix 34)
The sensor value is
a first sensor value acquired by a first sensor device;
a second sensor value acquired by a second sensor device different from the first sensor device;
Including,
In the image data determination step,
comparing the target image data obtained by imaging a part of the time-series data of the first sensor value with the failure representative image data obtained by imaging a part of the time-series data of the first sensor value;
comparing the target image data obtained by imaging a part of the time-series data of the second sensor value with the failure representative image data obtained by imaging a part of the time-series data of the second sensor value;
In the countermeasure acquisition step,
acquiring a first countermeasure in the case where a fault is determined in the time-series data of the first sensor value;
acquiring a second countermeasure in the case where a fault is determined in the time-series data of the second sensor value;
acquiring a third countermeasure in the case where a malfunction is determined in the time-series data of the first sensor value and the second sensor value;
A non-transitory computer-readable medium having stored thereon an information processing program according to any one of appendices 31 to 33, which causes a computer to execute the above-mentioned.
(Appendix 35)
A pseudo-fault image data acquisition step is further performed by the computer to acquire pseudo-fault image data by imaging the time-series data of the pseudo sensor values formed so as to indicate a pseudo-fault state of the customer device;
In the image data determination step,
determining a failure of the simulated failure state by comparing the simulated failure image data with the failure representative image data;
A non-transitory computer-readable medium having stored thereon an information processing program according to any one of appendices 31 to 34, which causes a computer to execute the above-mentioned.

1, 2, 3, 4 Information processing system 10 Failure representative image data storage device 10a Failure representative image data storage unit 11, 12 Failure representative image data 20 Countermeasure storage device 20a Countermeasure storage unit 30 Customer log data storage device 31 Time series data 40 Failure time storage device 50 Reproduction log data storage device 60 Customer image data storage device 70 Failure representative candidate image data storage device 80 Learning vector data storage device 90 Pseudo failure image data storage device 100, 200, 300, 400 Information processing device 110 Image data acquisition unit 120 Countermeasure acquisition unit 130 Image data conversion unit 140 Image data determination unit 150 Countermeasure notification unit 160 Image data vectorization unit 170 Image data extraction unit 180 Countermeasure input unit 190 Pseudo failure image data acquisition unit 400a Information processing unit A, B, C Element DP Display unit F1, F2 Window frame G11, G12, G13, G21, G22 Target image data LD Log data storage unit M0, M01, M02, M03 Customer device M1, M11, M12 Customer device NW Network SD Sensor device

Claims (10)

an image data acquisition means for acquiring failure representative image data obtained by imaging a part of time series data of sensor values acquired by a sensor device of the customer device when the customer device fails;
A countermeasure acquisition means for acquiring a countermeasure at the time of the failure;
an image data conversion means for converting a part of the time series data of the sensor value acquired by the sensor device of the customer device that is a target for failure determination into target image data;
an image data determination means for determining a failure of the customer device by comparing the representative failure image data with the target image data;
a countermeasure notification means for notifying the customer device of the countermeasure when the customer device is determined to be faulty;
Equipped with
The image data conversion means
a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data into a first time span, into an image while moving the first window frame back along the time axis of the time series data to a predetermined time at a first back time interval;
a plurality of second target image data pieces obtained by converting each second window frame into an image while moving the second window frame, which divides the time series data into a second time period greater than the first time period, back along the time axis of the time series data to the predetermined time at a predetermined second back time interval;
converting the target image data into
Information processing device. Further comprising an image data extraction unit for extracting the failure representative image data including the first failure representative image data and the second failure representative image data,
The image data extraction means includes:
extracting, from the time-series data, first failure representative candidate image data obtained by imaging a window portion divided by the time width of the first time including the time of the failure and immediately before the failure, the first failure representative candidate image data being first candidate image data obtained by imaging the window portion divided by the time width of the first time, when there are a first number or more pieces of first candidate image data having a first similarity or higher;
extracting, from the time-series data, second failure representative candidate image data obtained by imaging a window portion divided by the time width of the second time including the time of the failure and immediately before the failure, the second candidate image data obtained by imaging the window portion divided by the time width of the second time, when there are a second number or more of the second candidate image data having a second similarity or higher; the second failure representative candidate image data is extracted as the second failure representative image data;
The information processing device according to claim 1 . The failure representative image data and the target image data are vectorized into vector data by an image data vectorization unit.
the image data determination means determines whether the customer device is faulty by comparing the representative fault image data with the target image data using a vector distance between the representative fault image data and the target image data;
3. The information processing device according to claim 1 or 2. The sensor value is
a first sensor value acquired by a first sensor device;
a second sensor value acquired by a second sensor device different from the first sensor device;
Including,
The image data determination means
comparing the target image data obtained by imaging a part of the time-series data of the first sensor value with the failure representative image data obtained by imaging a part of the time-series data of the first sensor value;
comparing the target image data obtained by imaging a part of the time-series data of the second sensor value with the failure representative image data obtained by imaging a part of the time-series data of the second sensor value;
The countermeasure acquisition means includes:
acquiring a first countermeasure in the case where a fault is determined in the time-series data of the first sensor value;
acquiring a second countermeasure in the case where a fault is determined in the time-series data of the second sensor value;
acquiring a third countermeasure in the case where a fault is determined in the time-series data of the first sensor value and the second sensor value;
3. The information processing device according to claim 1 or 2 . a pseudo-fault image data acquisition means for acquiring pseudo-fault image data obtained by imaging a part of the time-series data of the pseudo sensor value formed so as to indicate a pseudo-fault state of the customer device;
The image data determination means
determining a fault in the pseudo-fault state by comparing the pseudo-fault image data with the fault representative image data;
3. The information processing device according to claim 1 or 2 . a customer device including a sensor device for acquiring a sensor value;
an information processing device connected to the customer device via a network;
a failure representative image data storage device that stores failure representative image data obtained by imaging a part of the time series data of the sensor value acquired by the sensor device of the customer device when the customer device fails;
a countermeasure storage device that stores countermeasures to be taken when the failure occurs;
Equipped with
The information processing device includes:
A failure representative image data acquisition means for acquiring the failure representative image data;
A countermeasure acquisition means for acquiring a countermeasure at the time of the failure;
an image data conversion means for converting a part of the time series data of the sensor value acquired by the sensor device of the customer device that is a target for failure determination into target image data;
an image data determination means for determining a failure of the customer device by comparing the representative failure image data with the target image data;
a countermeasure notification means for notifying the customer device of the countermeasure when the customer device is determined to be faulty;
having
The image data conversion means
a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data into a first time span, into an image while moving the first window frame back along the time axis of the time series data to a predetermined time at a first back time interval;
a plurality of second target image data pieces obtained by converting each second window frame into an image while moving the second window frame, which divides the time series data into a second time period greater than the first time period, back along the time axis of the time series data to the predetermined time at a predetermined second back time interval;
converting the target image data into the target image data including
Information processing system. a first customer device including a sensor device for acquiring a sensor value;
a second customer device including the sensor device for acquiring the sensor value;
an information processing device connected to the first customer device via a network;
Equipped with
The second customer device is
a failure representative image data storage means for storing failure representative image data that is an image of a part of the time series data of the sensor value acquired by the sensor device of the first customer device when the first customer device fails;
A countermeasure storage means for storing countermeasures to be taken when the failure occurs;
An image data acquisition means for acquiring the failure representative image data;
A countermeasure acquisition means for acquiring a countermeasure at the time of the failure;
an image data conversion means for converting a part of the time series data of the sensor value acquired by the sensor device of the second customer apparatus that is a target for failure determination into target image data;
an image data determination means for determining a failure of the second customer device by comparing the representative failure image data with the target image data;
a display means for displaying the countermeasure when the malfunction is determined;
having
The image data conversion means
a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data into a first time span, into an image while moving the first window frame back along the time axis of the time series data to a predetermined time at a first back time interval;
a plurality of second target image data pieces obtained by converting each second window frame into an image while moving the second window frame, which divides the time series data into a second time period greater than the first time period, back along the time axis of the time series data to the predetermined time at a predetermined second back time interval;
converting the target image data into the target image data including
Information processing system. A sensor device that acquires a sensor value;
a display means for displaying a countermeasure to be taken when a malfunction is determined;
Equipped with
an image data acquisition means for acquiring failure representative image data obtained by imaging a part of the time series data of the sensor value acquired by the sensor device of the apparatus when the apparatus fails;
A countermeasure acquisition means for acquiring a countermeasure at the time of the failure;
an image data conversion means for converting a part of the time series data of the sensor values acquired by the sensor device of the apparatus that is the subject of a failure judgment into target image data;
an image data determination means for determining a failure of the device by comparing the representative failure image data with the target image data;
a countermeasure notification means for notifying the device of the countermeasure when the device is determined to be faulty;
via a network to an information processing device having
The image data conversion means
a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data into a first time span, into an image while moving the first window frame back along the time axis of the time series data to a predetermined time at a first back time interval;
a plurality of second target image data pieces obtained by converting each second window frame into an image while moving the second window frame, which divides the time series data into a second time period greater than the first time period, back along the time axis of the time series data to the predetermined time at a predetermined second back time interval;
The information processing device converts the target image data into the target image data including
Customer equipment. A sensor device that acquires a sensor value;
An information processing means;
a display means for displaying a countermeasure to be taken when a malfunction is determined;
Equipped with
The information processing means includes:
an image data acquisition means for acquiring failure representative image data obtained by imaging a part of the time series data of the sensor value acquired by the sensor device of the apparatus when the apparatus fails;
a countermeasure acquisition means for acquiring the countermeasure when the failure occurs;
an image data conversion means for converting a part of the time series data of the sensor values acquired by the sensor device of the apparatus that is the subject of a failure judgment into target image data;
an image data determination means for determining a failure of the device by comparing the representative failure image data with the target image data;
having
The image data conversion means
a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data into a first time span, into an image while moving the first window frame back along the time axis of the time series data to a predetermined time at a first back time interval;
a plurality of second target image data pieces obtained by converting each second window frame into an image while moving the second window frame, which divides the time series data into a second time period greater than the first time period, back along the time axis of the time series data to the predetermined time at a predetermined second back time interval;
converting the target image data into
Customer equipment. an image data acquisition step of acquiring failure representative image data obtained by imaging a part of time series data of sensor values acquired by a sensor device of the customer device when the customer device is broken;
a countermeasure acquisition step of acquiring a countermeasure for the failure;
an image data conversion step of converting a part of the time series data of the sensor values acquired by the sensor device of the customer device that is a target for failure determination into target image data;
an image data determination step of determining whether the customer device has a malfunction by comparing the representative malfunction image data with the target image data;
a countermeasure notification step of notifying the customer device of the countermeasure when the fault is determined;
When you run the following on your computer:
In the image data conversion step,
a plurality of first target image data pieces obtained by converting a first window frame, which divides the time series data into a first time span, into an image while moving the first window frame back along the time axis of the time series data to a predetermined time at a first back time interval;
a plurality of second target image data pieces obtained by converting each second window frame into an image while moving the second window frame, which divides the time series data into a second time greater than the first time, back along the time axis of the time series data to the predetermined time at a predetermined second back time interval;
converting the target image data into the target image data including
An information processing program that causes a computer to execute

Images