JP7725217B2 - Anomaly detection method, detection device, and computer program - Google Patents

Description

The present invention relates to an anomaly detection method, detection device, and computer program for detecting comments related to anomalies in a specified game.

Elements such as scenarios, action, graphics, and background music in video games are becoming increasingly complex and involve an increasing amount of data, with a wide variety of options available. Patent Document 1 mentions that if the testing process for open-world-based games were to be left to humans, it would require a considerable number of man-months.

Japanese Patent Application Laid-Open No. 2020-108733

In online games, updates and event additions are often implemented from time to time. For example, when such updates or events are added, anomalies such as game crashes or bugs may occur in the game, and if the game's operators do not respond to these anomalies promptly, there is a risk that negative reactions to the game will spread, leading to users abandoning the game. Such anomalies can also occur at times other than when updates or events are added or around the same time, and when an anomaly-related incident occurs, it is necessary for the operators to recognize that the anomaly has occurred as soon as possible.

The present invention aims to provide an anomaly detection method, detection device, and computer program that can detect comments about anomalies in a specific game 24 hours a day, 365 days a year on a specific website (e.g., an electronic bulletin board or Social Networking Service (SNS)).

An anomaly detection method according to one embodiment of the present disclosure includes collecting data posted regarding a game application, inputting the collected data into an extraction model that has been trained to output data regarding an anomaly of the game application when the data is input, storing data from the collected data that is determined to be data regarding an anomaly based on the data output from the extraction model, inputting the stored data regarding an anomaly into a classification model that has been trained to sort the input data into a predetermined anomaly classification of bugs, malfunctions, and stops/crashes when the data is input, outputting classification data indicating the classification of the anomaly from the classification model, calculating a number of overlaps for each classification of the anomaly based on the output classification data indicating the classification of the anomaly, and storing the corresponding number of overlaps in association with each of the classification data indicating the classification of the anomaly.

A detection device according to one embodiment of the present disclosure includes a collection unit that collects data posted regarding game applications; a first memory unit that stores an extraction model trained to output data regarding anomalies in the game application when data is input, and a classification model trained to sort the input data into a pre-set anomaly classification of bugs, malfunctions, and stops/crashes ; a calculation unit that calculates the number of overlaps for each anomaly classification based on classification data indicating the anomaly classification output from the classification model; and a second memory unit that inputs the collected data into the extraction model and, based on the data output from the extraction model, stores data among the collected data that is determined to be data regarding anomalies and classification data indicating the anomaly classification in association with the corresponding number of overlaps.

A computer program according to one embodiment of the present disclosure causes a computer to collect data posted regarding a game application, input the collected data into an extraction model that has been trained so that when data is input, the data outputs data regarding an anomaly of the game application, store data from the collected data that is determined to be data regarding an anomaly based on the data output from the extraction model, input the stored data regarding an anomaly into a classification model that has been trained so that when data is input, the data is sorted into pre-set anomaly classifications of bugs, malfunctions, and stops/crashes , output classification data indicating the classification of the anomaly from the classification model, calculate a number of overlaps for each classification of the anomaly based on the output classification data indicating the classification of the anomaly, and store each of the classification data indicating the classification of the anomaly in association with the corresponding number of overlaps.

The anomaly detection method, detection device, and computer program disclosed herein automatically collect data posted by game users regarding the game application being detected, and use a trained model to store data that is likely to be related to an anomaly.

Data posted by users may be text data, image data, audio data, or video data. The data may also be a combination of two or more of text data, image data, audio data, and video data.

In one embodiment of the anomaly detection method disclosed herein, the model is trained to output, when data is input, a score corresponding to the likelihood that the data is data related to an anomaly in the game application, and determines whether the input data is data related to an anomaly based on the score.

In the anomaly detection method disclosed herein, stored data is associated with data on anomaly classifications using a trained model. Examples of classifications include "bug," "stop," and "crash."

In the anomaly detection method disclosed herein, data collection may be performed periodically. At the end of each period, a determination may be made as to whether the data is related to an anomaly, and classification may be performed for each period. The period may be variable and may be set.

An anomaly detection method according to one embodiment of the present disclosure creates a report containing a list of data related to stored anomalies and notifies the outside world of the creation of the report.

The anomaly detection method disclosed herein creates a report that can be viewed from a computer operated by an operator. Being able to view the report can speed up the process of responding.

In one embodiment of the anomaly detection method disclosed herein, the report includes a link for accessing each piece of data.

The anomaly detection method disclosed herein allows operators to access data (primary information) posted by game users through the reports that are created.

This disclosure makes it possible to detect comments about anomalies in a specific game on a specific website 24 hours a day, 365 days a year.

FIG. 1 is a schematic diagram of an anomaly detection system of the present disclosure. FIG. 2 is a block diagram showing the configuration of each device in the anomaly detection system. 10 is a flowchart illustrating an example of a data collection and extraction process performed by a detection device. 10 is a flowchart illustrating an example of a data classification processing procedure performed by the detection device. 10 is a flowchart illustrating an example of a procedure for a report creation process performed by a detection device. FIG. 10 is a diagram illustrating an example of a display on a client device. FIG. 10 is a diagram showing an example of the contents of a report display screen.

The present invention will be specifically described with reference to the drawings showing embodiments thereof.
The present invention extracts comments relating to anomalies in a specified game from a large number of comments posted on a website (such as an electronic bulletin board or SNS) where users can post and view their desired comments.
In this specification, an "anomaly" refers to an event that must be dealt with immediately in terms of the operation of a specific game, and relates to an abnormality in at least some elements of the game, such as an error, a malfunction, a bug, a game stopping, a game forced termination, etc. Therefore, for example, a comment regarding an anomaly in a specific game is a comment indicating that an error or bug has occurred in the specific game, or a comment calling attention to such an error or bug.

Figure 1 is a schematic diagram of an anomaly detection system 100 according to the present disclosure. The anomaly detection system 100 according to this embodiment detects comments related to anomalies in applications known as mobile games, which use smartphones or tablet devices as hardware. The hardware used to run the games is not limited to mobile devices, but may also be so-called game consoles such as home game consoles or arcade game machines, or personal computers. The detection targets are not limited to mobile game applications or game software (hereinafter referred to as game apps), but also include applications that use information processing devices such as smartphones or tablet devices as their main hardware.

The anomaly detection system 100 includes a detection device 1 and a client device 2 used by the creator of the game app to be detected. The detection device 1 is connected to a network that can be connected to the Internet. The detection device 1 transmits and receives data to and from the client device 2 via a server device connected to a network N either within the system or outside the system.

The detection device 1 collects data related to the game app to be detected via the Internet by crawling. The detection device 1 classifies the collected data and outputs a report for each classification. The detection device 1 continues crawling 24 hours a day without omission and stores the data in the memory unit 11. Note that crawling may be set to occur only during specified time periods. The detection device 1 automatically performs classification processing on the data stored sequentially. The detection device 1 creates a report based on the results of the classification processing and notifies the user. The timing of this creation and notification can be set as desired (for example, every hour, etc.).

Previously, for game apps that were the target of detection, staff would collect text posted on social media, blogs, etc. related to the game app, compile it into a report, and share it with relevant parties. Because this work was done manually, it was difficult to collect all posted text in a 24-hour period, given the cost and effort involved. Therefore, the collection and report creation were carried out at specific time periods. In this case, text posted outside of the collection time period could be omitted from the detection target, leading to omissions, or it could be collected at the next collection time period, resulting in a large time lag. Furthermore, the more users a game app has, the more difficult it can be to extract anomalies from the posted text. In other words, the more users a game app has, the greater the volume of illustrations, photos, and opinions (whether simple criticism or comments such as "it's fun") posted by fans of the game app, and these are not subject to anomaly detection and must be excluded from extraction.

In the anomaly detection system 100 of the present disclosure shown in FIG. 1, the detection device 1 automatically continues crawling the search target website and stores the data. The detection device 1 sequentially classifies the data collected by crawling, dividing it into time periods. This makes it possible to reduce the amount of posted data that is omitted from the collected data.

If the classified data includes, for example, text, audio, and video with the same content (purpose), it is advisable to select one of them as the representative data (name matching) and then compile it by adding the number of data items with the same content to that representative data.

The detection device 1 may extract data collected by crawling and classify the extracted data using a trained model. It is preferable to use a model that has already undergone supervised learning using data that has been manually extracted and classified in the past.

The detection device 1 creates a report from the classified data, stores it, uploads it to the server device, and notifies the client device. The report may be a list of the classified data or may be presented in a chart. Notification may be via data sharing, message exchange, or email, which are functions implemented by the server device.

It is desirable that reports displayed on the client device 2 be intuitive, easy to understand, and complete. The reports compiled in a spreadsheet may be saved sequentially in the data sharing application. It is desirable that the compiled report include direct links to each posted text.

The configuration and processing content of each device used to realize this anomaly detection system 100 will now be described in detail. Figure 2 is a block diagram showing the configuration of each device in the anomaly detection system 100. The detection device 1 includes a processing unit 10, a memory unit 11, and a communication unit 12.

In the following explanation, the detection device 1 is described as being composed of a single server computer, but it may also be configured as multiple server computers connected via a network for distributed processing. The detection device 1 may be a cloud-based server that can be connected to from an information processing device via a public network, or it may be an on-premise server that is connected to each client device 2 via a virtual or physical private network.

The processing unit 10 is a processor that uses a CPU (Central Processing Unit) and/or a GPU (Graphics Processing Unit). Based on the detection program 1P stored in the memory unit 11, the processing unit 10 collects data related to the target game app and performs extraction and classification.

The storage unit 11 uses non-volatile memory such as a hard disk or SSD (Solid State Drive). The storage unit 11 stores data referenced by the processing unit 10. The storage unit 11 stores a detection program 1P. The storage unit 11 stores a trained extraction model 11M used for extraction and a trained classification model 12M used for classification. The detection program 1P may be a detection program 9P stored on the recording medium 9 that has been read by the processing unit 10 and copied to the storage unit 11. The extraction model 11M or classification model 12M may also be an extraction model 91M or classification model 92M stored on the recording medium 9 that has been read by the processing unit 10 and copied to the storage unit 11.

The memory unit 11 stores data related to anomalies extracted and classified by the process described below. The memory unit 11 that stores this data related to anomalies may be an external storage medium such as a flash memory, and may be removable from the detection device 1.

The communication unit 12 realizes data communication via the network N. Specifically, the communication unit 12 is, for example, a network card. The processing unit 10 accesses data on the Internet via the communication unit 12 and performs crawling.

The client device 2 is, for example, a laptop personal computer. The client device 2 may also be a desktop personal computer, a tablet terminal, or a so-called smartphone.

The client device 2 includes a processing unit 20, a memory unit 21, a communication unit 22, a display unit 23, and an operation unit 24. The processing unit 20 is a processor that uses a CPU and/or GPU. Based on a client program 2P stored in the memory unit 21, the processing unit 20 causes the display unit 23 to display a report obtained from the detection device 1 via the server device.

The storage unit 21 uses non-volatile memory such as a hard disk, flash memory, or SSD. The storage unit 21 stores data referenced by the processing unit 20. The storage unit 21 stores the client program 2P.

The communication unit 22 realizes data communication via the network N. Specifically, the communication unit 22 is, for example, a network card. The processing unit 20 can acquire data from the detection device 1 via the server device using the communication unit 22.

The display unit 23 is a display such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display unit 23 displays a screen containing information based on data stored in the memory unit 21 or data provided by the detection device 1. The display unit 23 may be a display with a built-in touch panel.

The operation unit 24 is a user interface such as a keyboard and pointing device that can input and output data to and from the processing unit 20. The operation unit 24 may be a voice input unit. The operation unit 24 may be a touch panel built into the display unit 23. The operation unit 24 may be physical buttons.

Figure 3 is a flowchart showing an example of the data collection and extraction process performed by the detection device 1. The detection device 1 repeatedly executes the process shown below at a predetermined timing. The predetermined timing may occur periodically, or may occur when a predetermined condition is met.

The processing unit 10 accesses a website (such as a specified community site, bulletin board site, SNS, or posting site) set as a detection target via the network N and acquires data (text, audio, images, or video) posted during a target period related to the game app to be detected (step S101). The target period is the period from the arrival of the current timing back to the arrival of the previous timing.

The target period may be set by the operator via the client device 2. This shortens the time between a user posting and the creator receiving the information, allowing for more immediate anomaly detection. If the creator anticipates an increase in the frequency of anomalies, such as during a limited-time event or immediately after a major update, it is possible to change the division of this period to increase the frequency of detection.

In step S101, if the site is a predetermined community site, the processing unit 10 accesses each "group" related to the game app. If the site is a bulletin board site, the processing unit 10 accesses each so-called "board," "genre," "thread," or "topic." If the site is a social networking site or posting site, the processing unit 10 acquires data extracted using keywords such as a hashtag for the game name or the names of characters appearing in the game. The processing unit 10 may crawl all data related to the target game app, or may select and acquire data based on keywords specified in advance at the time of step S101.

The processing unit 10 selects data from the acquired data (step S102) and provides it to the extraction model 11M (step S103).

In step S103, the processing unit 10 may convert the selected data into text, perform morphological analysis, and then provide the text as a word list (vector) to the extraction model 11M. If the data is audio, the processing unit 10 may perform speech recognition and then morphological analysis. If the data is an image, the processing unit 10 may perform character recognition on the characters included in the image, or convert the metadata into text and then perform morphological analysis. If the data is a video, the processing unit 10 may perform both character recognition on the characters included in the image and speech recognition on the audio.

The processing unit 10 determines whether the data relates to an anomaly such as a bug or malfunction based on the score from the extraction model 11M (step S104).

In step S104, the processing unit 10 obtains from the extraction model 11M a score corresponding to the likelihood that each word in the input word list is an anomaly-related data. The extraction model 11M uses a neural network. For example, it is a model using a TextCNN (Convolutional Neural Network). A recurrent neural network (RNN) that learns by taking time series elements into account, or a recurrent neural network (LSTM: Long short-term memory / GRU: Gated Recurrent Unit), etc., may also be used. The extraction model 11M may be trained in advance using previous manual extraction results, or may be trained using newly created training data. For example, the extraction model 11M may be trained by assigning a high likelihood that words such as "bug," "glitch," "crash," "problem," "stop," and "strange" are anomalies. The processing unit 10 sums the scores obtained from the extraction model 11M for each word and calculates a score for whether the data is an anomaly-related data relative to the entire data. The extraction model 11M for determining whether data is related to an anomaly is not limited to this, and any known method for analyzing text may be used. In steps S103 and S104, the processing unit 10 may determine whether the data is negative using an extraction model 11M that outputs a score corresponding to the likelihood that the data is negative (or positive).

If the processing unit 10 determines that the data is related to an anomaly (S104: YES), it stores the data in the memory unit 11 (step S105) and proceeds to step S106. If the processing unit 10 determines that the data is not related to an anomaly (S104: NO), it proceeds to step S106.

In step S105, the processing unit 10 may store a word list of the data morphologically analyzed in step S103. In step S105, the processing unit 10 may store link data that allows direct access to the original data (site, post) of the acquired data.

The processing unit 10 determines whether the process of providing all of the data acquired in step S101 to the extraction model has been completed (step S106), and if it is determined that the process has been completed (S106: NO), the processing returns to step S102.

If it is determined that all of the acquired data has been applied to the extraction model (S106: YES), the processing unit 10 terminates the process.

The processing unit 10 may execute the processing procedure shown in the flowchart of Figure 3 for each predetermined site. The processing procedure shown in the flowchart of Figure 3 extracts data related to anomalies in the target game app, and the results are stored in the memory unit 11.

Figure 4 is a flowchart showing an example of a data classification processing procedure performed by the detection device 1. The detection device 1 may execute the processing shown below immediately after the detection device 1 completes its data collection and extraction processing. The execution timing of the following processing and the execution timing of the data collection and extraction processing of Figure 3 may be independent of each other, or may be executed in parallel.

The processing unit 10 reads extracted data related to anomalies for the game to be detected from the memory unit 11 (step S201).

The processing unit 10 provides the read data to the classification model 12M (step S202). In step S202, the processing unit 10 may convert the read data into text, perform morphological analysis, and then provide the data to the classification model. If a morphologically analyzed word list is stored, the word list may be provided. The classification model 12M may be trained in advance based on previous manual classification results. Examples of classifications include "bug," "glitch," "stop/crash," and "improvement request." The processing unit 10 obtains from the classification model 12M a score indicating the likelihood that each word in the input word list belongs to one of the pre-set classifications. The classification model 12M may employ a known method for classifying text. The classification model 12M may be trained to output a score corresponding to the level of severity along with the classification.

The processing unit 10 associates the classification data indicating the classification of the given data based on the score from the classification model 12M with the read data and stores it in the memory unit 11 (step S203). The processing unit 10 determines which classification the entire data belongs to based on the obtained score.

The processing unit 10 determines whether any unclassified data remains in the memory unit 11 (step S204). If the processing unit 10 determines that any unclassified data remains (S204: YES), the processing returns to step S201, and the processing continues by reading the unclassified data.

If it is determined in step 204 that no items remain (S204: NO), the processing unit 10 terminates the classification process.

The processing procedure shown in the flowchart in Figure 4 classifies data extracted from specified sites and SNS as being highly likely to be data related to anomalies. The classified data (text, audio, images, or videos) is stored in association with the classified data in the memory unit 11. This data can be read and displayed from the client device 2 along with the classified data, allowing producers to more quickly identify defects, bugs, etc. Each piece of classified data is stored in association with a link to access the original data for that data. This allows the client device 2 to read the data stored in the memory unit 11, making it easy for operators to check the primary information posted by users of the game app being detected.

Figure 5 is a flowchart showing an example of a report creation process performed by the detection device 1. The detection device 1 may execute the process described below immediately after the detection device 1 completes its data classification process. The execution timing of the following process, the execution timing of the data collection and data extraction process in Figure 3, and the execution timing of the data classification process in Figure 4 may be independent of each other, or may be executed in parallel.

The processing unit 10 reads out, for each category, the classified data associated with classification data from the storage unit 11 (step S301). In step S301, the processing unit 10 reads out a word list or the original data (text, audio, image, or video). In the case of audio, image, or video, the processing unit 10 may read out a text version of the data.

The processing unit 10 counts the number of data items for each category (step S302) and stores the number of data items in association with the category data and time information (step S303). The processing unit 10 consolidates duplicate data items for each category (step S304). Step S304 is optional. In step S304, the processing unit 10 selects one piece of data for each category and determines, one by one, whether the content of the selected data and other data items in the same category overlap, i.e., whether the text overlaps. If the degree of overlap is equal to or greater than a predetermined level, the number of overlaps for the selected data is incremented. The selected data and the number of overlaps for that data are stored in association with the data's identification data. The processing unit 10 repeatedly selects data and consolidates duplicate data items.

The processing unit 10 creates a report using document data that lists the number of data items for each category, time information, the text of the selected data, and the number of duplicates (step S305). In step S305, the processing unit 10 creates spreadsheet data, document data, email, or specific HTML data. The report creation step may be realized by equipping the detection device 1 with a display unit, and automating steps such as reading data by category from the memory unit 11 and displaying the list in, for example, a spreadsheet program, and sorting the displayed list using RPA (Robotic Process Automation).

The processing unit 10 notifies the client device 2 that the report has been created (step S306) and ends the process.

In step S306, the processing unit 10 notifies the user by a message, email, or the like that includes a link to the report. The recipients of the notification may be set in advance. For example, operator identification data for the operators to be notified for each category may be stored. The operator identification data may be an email address, an account in a data sharing application, or a message application. The operator identification data may also be a community ID or team ID in a data sharing application.

The report notification may be delivered on an external storage medium containing the extracted and classified data. For example, the data may be stored verbatim on a portable flash memory.

The report creation process in Figure 5 is not required. If it is determined that classification into all data has been completed during the classification process procedure shown in the flowchart in Figure 4 (S204: NO), the processing unit 10 of the detection device 1 may notify the client device 2 in step S306.

Anomaly detection using the processing procedures shown in the flowcharts of Figures 3 to 5 will be described using a specific example. The anomaly detection system 100 detects comments related to anomalies in a specific game app. The creator of the specific game app, for example, stores the address of a community site related to the game app in advance in the memory unit 11 of the detection device 1. The community site for the specific game app is a message posting site and bulletin board for each game on a game strategy site. These posting sites and bulletin boards may be categorized by event or strategy theme. The creator may specify a URL specifying a specific "board" or "genre" on the bulletin board site and store it in the memory unit 11. The specification is not limited to a "board" or "genre," but may also be a search keyword. The creator may specify a search keyword for the SNS and store it in the memory unit 11. The detection device 1 has server functionality and may accept and store these specifications from the client device 2. When specifying search keywords, abbreviations, etc. used by the user must also be specified.

In the specific example described below, we will assume that the abbreviation for the game app is "ABC," and that a message has been posted with image data attached saying, "I think this is a bug." The specific example will also assume that messages such as "It froze" and "The display is strange" have been posted to a community site.

The processing unit 10 of the detection device 1 acquires data using the stored address, URL, and search keyword (S101). For example, the processing unit 10 accesses a specified address within a site for attacking the community site "ABC" and acquires the posted text. If image data has been uploaded along with the text, the processing unit 10 also acquires the image data. At this time, the processing unit 10 narrows down the data acquired to data posted during the target period on the specified community site without sifting through it. By acquiring data without searching using search keywords, the processing unit 10 can minimize data missed by the search. For example, the processing unit 10 may acquire data posted by users from a social networking site using "ABC" as a search keyword.

As mentioned above, the cycle for data acquisition by the detection device 1 can be set by the operator from the client device 2. If the cycle is set shorter than the specified initial value, the time from user posting to the completion of classification can be shortened. During periods when the possibility of anomalies occurring is low, such as when no events or updates are scheduled for the game application or there are no OS updates for the hardware, the cycle can be set longer than the specified initial value.

The processing unit 10 provides the word list obtained by morphologically analyzing the acquired data to the extraction model 11M (S103). As described above, the extraction model 11M outputs a score for each word included in the word list indicating the likelihood that the post or comment is related to an anomaly. The processing unit 10 totals the scores for each word included in the word list to calculate a score indicating the likelihood that the entire data is related to an anomaly. At this time, it is preferable to assign weights to specific words such as "bug," "glitch," "crash," and "failure," so that posts and comments containing these words can be determined to be data related to an anomaly with a high probability. The extraction model 11M itself may be trained by assigning large weights to these words. The score may also be divided by the number of words included in the word list to calculate the score as a numerical value, such as the occurrence probability of words that are likely to be determined to be data related to an anomaly, rather than the total.

The processing unit 10 determines whether the data is related to an anomaly based on whether the score calculated for the entire data is equal to or greater than a predetermined value (S104). For example, if a post with attached image data saying "I think this is a bug" is posted to a community site, morphological analysis will analyze it as "This/bug/is/to/I think." No scores are calculated for particle words such as "this," "is," or "to," but a score indicating the likelihood that the noun word "bug" is related to an anomaly is calculated. By assigning a large weight to "bug," the processing unit 10 determines, based on the score calculation results, that the text of the post saying "I think this is a bug" and the attached image data are data related to an anomaly.

The processing unit 10 may perform character recognition on the attached image data before providing it to the extraction model 11M. The image data may be a game screen or an illustration that recreates a game screen. If it is a game screen, the processing unit 10 can convert the results of the character recognition into text, such as the names of enemy and ally characters, lines, status values, conditions, skill names for special effects, and names of selected techniques included in the game screen, and use this as related data. Even if the data attached to the note is not image data but moving image data or audio data, the processing unit 10 can similarly convert this into text and use it as related data.

The processing unit 10 may also provide the associated data, which is a text version of the attached data, to the extraction model 11M in step S103. However, if it is treated at the same level as written text, there is a possibility that it will be recognized as data related to an anomaly based on negative comments. It is recommended that the text resulting from image recognition not be provided to the extraction model 11M, and that the associated data obtained by image recognition be used for subsequent classification.

The processing unit 10 stores the target data determined to be data related to an anomaly (S105). As described above for step S105, the processing unit 10 stores the acquired text of the post "I think this is a bug," the image data attached to the text, and the word list. The processing unit 10 may also store a URL for accessing the post itself, including the text "I think this is a bug" and the attached image data. This may be stored as cache data rather than as a URL for direct access. If image data is attached, the processing unit 10 may also store the text recognized from the image data. As described above, this is for use during classification.

During the classification process, the processing unit 10 reads data stored in the memory unit 11 that includes the text "I think this is a bug." The processing unit 10 may read the original text or text obtained by image recognition or voice recognition. If a word list after morphological analysis is stored, the processing unit 10 may read this.

The processing unit 10 provides the classification model 12M with a word list after morphological analysis of the read data (S202), and determines the classification based on the score for each classification for each word obtained from the classification model 12M. When the data "I think this is a bug" is provided to the classification model 12M, the processing unit 10 can obtain a score for the word "bug" that indicates a high probability that it is classified as a "bug". For other words, such as "I think this is a bug", scores may not be calculated because they are generic words. The processing unit 10 cannot obtain scores for classifications other than "bug", and associates the data "I think this is a bug" with classification data (identification number) indicating the "bug" classification and stores it. Note that the classification process does not have to be limited to processes that use the classification model 12M.

In step S202, the processing unit 10 may further classify the broad category of "bug" into more detailed categories based on associated data such as character names and lines obtained from the attached image data associated with the data.

As a result of the classification process, the memory unit 11 stores the extracted data and classification data indicating the classification of the data for each period.

In a specific example, the processing unit 10 creates a report. The processing unit 10 reads the classified data by category (S301) and compiles it (S304). The processing unit 10 may compile posts such as "I think this is a bug" and posts such as "Isn't this a bug?", "It's a bug," "I found a bug," and "A bug has occurred." The processing unit 10 may determine that the degree of overlap is high when, among the data associated with the classification data of "bug," a specific word, for example, "bug," is common and the words other than the word "bug" are general-purpose words. The processing unit 10 may determine whether the words other than the specific word, "discovery" and "think," are similar to each other using a thesaurus, co-occurrence dictionary, etc., and, if they are similar, process them to increase the degree of overlap. The processing unit 10 may also determine overlap in meaning using known language processing.

The processing unit 10 creates a report of the counting and/or summarizing results of step S302 (S305). In a specific example, the processing unit 10 creates the report in both spreadsheet data and document data. The processing unit 10 creates a report in spreadsheet data that describes, by category, the text of the classified data extracted in each row and link data for accessing that data. Spreadsheet data here means data in a format that can be viewed in a spreadsheet application. The processing unit 10 creates the report in document data that describes, for each category, the text of the summarized data and the number of duplicates. The document data does not need to include link data, etc. It is preferable that the document data include multiple written texts and image data of representative data of the summarized data.

The processing unit 10 uploads the created report to a file sharing application. The processing unit 10 notifies the developer by sending a message including link data to the report (spreadsheet data, document data, etc.) in the file sharing application to the developer's team ID in the message application. The client device 2 can receive the notification via the message application. The processing unit 10 may change the identification data identifying the operator to be notified for each category. The processing unit 10 may send a message including a link to spreadsheet data containing links to each data to the identification data of the operator who is the developer, and may send a message including a link to the document data to the identification data of the operator who is the planner. The team ID and community ID may be separately notified.

A display example on the client device 2 in this specific example will be described with reference to the drawings. FIG. 6 is a diagram showing an example display on the client device 2. FIG. 6 shows a home screen 230 of the message application. FIG. 6 shows an example of the home screen 230 displayed when a developer operator among the creators logs in with their account. When the message application is launched with the developer operator's account, as shown in FIG. 6, the home screen 230 includes a list area 231 of teams, themes, etc. for each project in which the account is participating, and a display area 232 for messages for a team or theme selected from the list.

Icon 233 indicates that one notification has been received for the team (theme) "Anomaly Report" in the project "ABC" included in the list area 231.

Display area 232 displays the contents of messages exchanged with other operators in the selected team or theme. In the display example of Figure 6, a message is displayed indicating that an "anomaly report" has been uploaded from detection device 1. Display area 232 displays icon 234 indicating a link to the report. When icon 234 is selected by operation unit 24, a report display screen is displayed on display unit 23 based on the address to the report in the data sharing application associated with icon 234.

Figure 7 is a diagram showing an example of the contents of the report display screen 235. The report display screen 235 displays a report in table format using a spreadsheet application launched within the data sharing application. In the example of Figure 7, a list of data text is shown for each category. The number shown in the cell next to the text is the number of data with the same content, i.e., the number of duplicates. When the number of duplicates is selected using the operation unit 24, a list of all data may be displayed.

In the example in Figure 7, the categories "Bug" and "Stop" are displayed. Corresponding to the "Bug" category, data on posts such as "I think this is a bug" and the number of duplicates with the same meaning are displayed. In the example in Figure 7, text on posts with the same meaning as "I think this is a bug," such as "I found a bug" and "A bug has occurred," is displayed on separate lines. Each line corresponding to data with the same meaning contains a link to directly view each piece of data, and text on related data stored in association with each piece of data.

The "Bug" category also displays data on posts stating "The display is strange" and the number of duplicates. Data on posts with the same meaning as "The display is strange" includes data on posts stating "The display is bugged."

The "Stopped" category displays data on posts that say "It froze" and the number of duplicates.

In the example of Figure 7, data is sorted and displayed for each category in descending order of the number of duplicates. The more posts with the same content, the sooner the operator can check them, shortening the time lag from the occurrence of an anomaly. Regarding categories, categories with a higher number of detections may also be displayed at the top. Data may also be sorted and displayed by the level of severity determined.

In this way, operators can collectively review all of the data that has been automatically and thoroughly crawled by the detection device 1. By displaying data by classification, operators can collectively review the content that needs to be addressed. Responding to anomalies promptly is effective in preventing active users of game apps that use smartphones as hardware from abandoning the app. Although identifying anomalies requires significant costs, reviewing all posts on websites related to the game app being detected requires significant costs. However, reviewing data automatically collected, extracted, and classified by the detection device 1 as a report reduces manpower and time costs. Automatic crawling is possible 24 hours a day, 365 days a year, reducing the risk of a decline in appeal due to anomalies.

As shown in Figure 7, the report also displays the number of duplicates, allowing users to understand the number of recognized anomalies compared to others. Each piece of data is associated with a link to access the original post or text, allowing operators to check the primary information and more quickly deduce the cause of the anomaly from the preceding and following information, enabling them to take action and make arrangements more quickly.

For data detected as anomaly-related data (comments) as shown in FIG. 7, an operator may point out data that is not an anomaly and provide feedback to the detection device 1. The detection device 1 may also re-train the extraction model 11M using data that is not an anomaly but is determined by the extraction model 11M to be data related to an anomaly.

Specific community sites, boards, genres, search keywords, etc. to be crawled, as well as the period, may be specified to the detection device 1 via the message application shown in FIG. 6 or the data sharing application shown in FIG. 7. For example, the detection device 1 connects to the data sharing application using an account assigned to the detection device 1, references a stored configuration file, reads the search keywords, etc. contained in the configuration file, stores them in the memory unit 11, and uses them.

The embodiments disclosed above are illustrative in all respects and are not restrictive. The scope of the present invention is defined by the claims, and includes all modifications that are equivalent in meaning to and within the scope of the claims.

100 Anomaly detection system 1 Detection device 10 Processing unit 11 Storage unit 1P Detection program 2 Client device 20 Processing unit 23 Display unit 24 Operation unit 2P Client program

Claims (5)

Collect data posted about the game application,
inputting the collected data into an extraction model that has been trained to output data related to an anomaly of the game application when the data is input;
storing data determined to be data relating to an anomaly among the collected data based on the data output from the extraction model;
inputting the stored data relating to an anomaly into a classification model that has been trained to classify the data into a preset anomaly classification of bug, malfunction, or stop/crash when the data is input, and outputting classification data indicating the classification of the anomaly from the classification model;
calculating the number of overlaps for each anomaly classification based on the classification data indicating the classification of the anomaly output;
The anomaly detection method stores each of the classification data indicating the classification of the anomaly and the corresponding number of overlaps in association with each other. periodically collecting posted data relating to said game application;
The anomaly detection method according to claim 1 , wherein collected data is input into the extraction model for each period at the end of each period. Generate a report that lists the data about the stored anomalies,
3. The anomaly detection method according to claim 1, further comprising the step of notifying an external party of the creation of the report. a collection unit that collects data posted regarding the game application;
a first storage unit that stores a trained extraction model that, when data is input, outputs data related to an anomaly of the game application for the input data, and a trained classification model that, when data is input, classifies the input data into a preset anomaly classification of bugs, malfunctions, and stoppages/crashes ;
a calculation unit that calculates the number of overlaps for each classification of the anomaly based on classification data indicating the classification of the anomaly output from the classification model;
A detection device comprising: a second memory unit that inputs collected data into the extraction model, and stores, based on the data output from the extraction model, data among the collected data that is determined to be data related to anomalies, and classification data that indicates the classification of the anomalies, in association with the corresponding number of overlaps. On the computer,
Collect data posted about the game application,
inputting the collected data into a trained extraction model such that, when the data is input, the model outputs data related to an anomaly of the game application;
storing data determined to be data relating to an anomaly among the collected data based on the data output from the extraction model;
inputting the stored data relating to an anomaly into a classification model that has been trained to classify the data into a preset anomaly classification of bug, malfunction, or stop/crash when the data is input, and outputting classification data indicating the classification of the anomaly from the classification model;
calculating the number of overlaps for each anomaly classification based on the classification data indicating the classification of the anomaly output;
A computer program that executes a process of storing each of the classification data indicating the classification of the anomaly in association with the corresponding number of overlaps.