JP6946530B2 - Methods and equipment to provide verification crowdsourcing work based on production work - Google Patents

Description

The present invention relates to a method and an apparatus for providing a verification crowdsourcing operation based on a production operation.

In machine learning, more important than learning methods and algorithms is data for learning. Therefore, it is very important to secure a lot of good quality learning data in the machine learning field.

In order to obtain training data, it is necessary to collect source data or perform data annotation work. Where to study or develop machine learning with respect to source data collection or data annotation work, there may be a labor force to perform source data collection or data annotation work.

However, since source data collection or data annotation work is often a work that anyone can do without specialized knowledge, data is distributed to a large number of workers via the Internet in order to reduce labor costs. , Platforms that pay a fee according to the collection of source data or data annotation work have appeared. For example, there is Amazon's Mechanical Turk.

However, in the case of an existing platform, there is a problem that a lot of cost and time are required in the process of inspecting whether the worker has sincerely collected the source data or performed the data annotation work. Therefore, it is required to develop a technique for improving the reliability of learning data by efficiently collecting source data or accepting data annotation work while maintaining the advantages of the existing platform.

Korean Patent Publication No. 10-2012-0149930 (2012.12.20)

The present invention has been made in view of the above circumstances, and an object of the present invention is to acquire and provide a verification crowdsourcing work provided in a verification method of a production crowdsourcing work from the production work.

The problem to be solved by the present invention is not limited to the problem mentioned above, and other problems not mentioned can be clearly understood by ordinary engineers from the following description.

The method of providing the verification crowdsourcing work based on the production work according to the embodiment of the present invention for solving the above-mentioned problems is a method performed by a computer, and a verification crowdsourcing work pool is constructed. The production stage and the result of the production crowdsourcing group work are received from the worker terminal, and the production crowdsourcing group work is composed of one or more production crowdsourcing work. Crowdsourcing group work result reception stage and one or more verification crowdsourcing work corresponding to the production crowdsourcing group work are selected from the verification crowdsourcing work pool and transmitted to the worker terminal. Based on the stage, the stage of receiving the result of the one or more verification crowdsourcing work from the worker terminal, and the result of the one or more verification crowdsourcing work, the production crowdsourcing group work The verification crowdsourcing work pool includes a plurality of verification crowdsourcing operations, including a stage of determining whether or not the result is valid, and a stage of constructing the verification crowdsourcing work pool is a crowdsourcing work project. From the results of one or more production crowdsourcing operations that have already been performed in real time during the process of , The stage of generating the production work platform verification crowdsourcing work based on the result of the acceptance-passing crowdsourcing work satisfying the predetermined conditions among the extracted results of the acceptance-passing crowdsourcing work, and the generated production The work infrastructure verification crowdsourcing work is included in the verification crowdsourcing work pool.

As an embodiment, the predetermined condition corresponds to a predetermined order based on the time required for acceptance inspection.

As an embodiment, the steps for generating the verification crowdsourcing work include a step of extracting the result of the acceptance-passing crowdsourcing work satisfying a predetermined condition from the extracted result of the acceptance-passing crowdsourcing work and the step of extracting the result of the acceptance-passing crowdsourcing work and the predetermined condition. The stage of requesting one or more inspectors different from the inspector to re-inspect the result of the crowdsourcing work that satisfies the above, and the re-inspection as a result of the re-inspection that has passed the acceptance by all of the one or more inspectors. It includes the stage of generating the crowdsourcing work for verification of the production work platform based on the result of the crowdsourcing work that passed the acceptance inspection.

As an embodiment, the step of transmitting one or more verification crowdsourcing operations to the worker terminal excludes the verification crowdsourcing operation already provided to the same worker terminal as the worker terminal. The above verification crowdsourcing work is transmitted to the worker terminal.

As an embodiment, the specific verification crowdsourcing work in the verification crowdsourcing work pool is generated based on the result of one or more production crowdsourcing work received from the same worker terminal as the worker terminal. If so, at the stage of transmitting one or more verification crowdsourcing operations to the worker terminal, the worker performs one or more verification crowdsourcing operations excluding the specific verification crowdsourcing operation. Send to the terminal.

As an example, in the stage of constructing the verification crowdsourcing work pool, the verification crowdsourcing work generated in advance by the administrator is included in the verification crowdsourcing work pool before the execution of the crowdsourcing work project. Including that further.

As an embodiment, in the step of selecting from the verification crowdsourcing work pool and transmitting it to the worker terminal, after the pre-generated verification crowdsourcing work is exhausted, the production work platform verification crowdsourcing The work is transmitted to the worker terminal.

The method of providing the verification crowdsourcing work based on the production work according to the embodiment of the present invention for solving the above-mentioned problems depends on whether the result of the production crowdsourcing group work is valid or not. The stage of deciding whether or not to perform acceptance inspection for the result of the production crowdsourcing group work is further included, and the stage of deciding whether or not to perform acceptance inspection for the result of the production crowdsourcing group work is the stage of determining whether or not to perform acceptance inspection. If the result of the work matches the predetermined answer, it is decided to inspect the result of the relevant production crowdsourcing group work, and the result of the verification crowdsourcing work is the predetermined answer. If it does not match, the result of the corresponding production crowdsourcing group work is deleted and it is decided not to perform acceptance inspection.

A computer program that provides a crowdsourcing operation for verification based on a production operation according to another embodiment of the present invention for solving the above-mentioned problems is combined with a computer that is hardware, and any one of the above methods. Stored in the medium to perform the method of the term.

The crowdsourcing work providing device for verification based on the production work according to another embodiment of the present invention for solving the above-mentioned problems is a crowdsourcing work pool construction unit for verification that constructs a crowdsourcing work pool for verification. , The result of the production crowdsourcing group work is received from the worker terminal, and the production crowdsourcing group work is composed of one or more production crowdsourcing work. The production crowdsourcing group. The work result receiving unit and one or more verification crowdsourcing operations corresponding to the production crowdsourcing group work are selected from the verification crowdsourcing work pool and transmitted to the worker terminal. Based on the work transmission unit, the verification crowdsourcing work result receiving unit that receives the result of the one or more verification crowdsourcing work from the worker terminal, and the result of the one or more verification crowdsourcing work. , The crowdsourcing work pool for verification includes a plurality of crowdsourcing work for verification, and the crowdsourcing for verification includes a validness judgment unit for judging whether or not the result of the crowdsourcing group work for production is valid. The work pool construction department extracts the results of crowdsourcing work that has passed the acceptance of the inspector from the results of one or more production crowdsourcing work that has already been performed in real time while the crowdsourcing work project is in progress. Then, among the results of the extracted crowdsourcing work that passes the acceptance inspection, the crowdsourcing work for verification of the production work platform is generated based on the result of the crowdsourcing work that passes the acceptance inspection that satisfies the predetermined conditions, and the generated production work platform is generated. The verification crowdsourcing work is included in the verification crowdsourcing work pool.

Other specific matters of the present invention are included in the detailed description and drawings.

According to the present invention, even if all the pre-generated verification crowdsourcing work is provided to the worker, the verification crowdsourcing work is continuously updated based on the actual work, so that the worker can continue. And can provide new verification crowdsourcing work.

Further, according to the present invention, by providing the production crowdsourcing work data as the verification crowdsourcing work, the verification crowdsourcing work can be utilized as the evaluation data of the work more suitable for the project.

Further, according to the present invention, among the crowdsourcing operations that have passed the acceptance inspection for a specific inspector, the work that has a long working time and has passed all the acceptance inspections from other inspectors is for verification. By providing it as a crowdsourcing work, even if the difficulty level is high, the evaluation result for the worker can be calculated more accurately through the highly accurate verification crowdsourcing work.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned above can be clearly understood by ordinary engineers from the following description.

It is a conceptual diagram of the crowdsourcing service by one Example of this invention. It is a flowchart explaining the progress process of the crowdsourcing project by one Example of this invention. It is a schematic sequence diagram about the method of providing the crowdsourcing work for verification based on the production work by one Example of this invention. It is a schematic illustration about the crowdsourcing work for the image which contains an object. It is a figure explaining the method of constructing the crowdsourcing work pool for verification. It is a figure explaining the method of providing the production crowdsourcing work and the verification crowdsourcing work. It is an example diagram (1) explaining the method of acquiring the crowdsourcing work for verification from the result of the production crowdsourcing work. It is an example diagram (2) explaining the method of acquiring the crowdsourcing work for verification from the result of the production crowdsourcing work. When the result of the production crowdsourcing work is not valid, it is an example diagram which provides the production crowdsourcing work and the verification crowdsourcing work after this. When the result of the production crowdsourcing work is valid, it is an example diagram which provides the production crowdsourcing work and the verification crowdsourcing work after that. It is a schematic block diagram of the crowdsourcing work providing apparatus for verification based on the production work by another Example of this invention.

The advantages and features of the present invention, and the methods for achieving them, will be clarified with reference to the examples described in detail with the accompanying drawings. However, the present invention is not limited to the examples disclosed below, and can be realized in various forms different from each other. However, the present embodiment is provided to complete the disclosure of the present invention and to allow ordinary engineers in the technical field to which the present invention belongs to fully understand the scope of the present invention, and the present invention is claimed. It is only defined by the scope.

The terms used herein are for illustration purposes only and are not intended to limit the invention. In the present specification, the singular form also includes the plural form unless otherwise specified. As used herein, "comprises" and / or "comprising" does not preclude the presence or addition of one or more other components other than those mentioned. The same drawing reference throughout the specification indicates the same component, where "and / or" includes each and any combination of one or more of the mentioned components. It goes without saying that even if "first", "second", etc. are used to describe various components, these components are not limited by these terms. These terms are used simply to distinguish one component from the other. Therefore, it goes without saying that the first component referred to below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms used herein (including technical and scientific terms) are used in the sense commonly understood by ordinary engineers in the technical field to which the present invention belongs. Also, commonly used terms defined in dictionaries are not ideally or over-interpreted unless explicitly specifically defined.

As used herein, production (actual) work typically refers to the direct work of generating or collecting data that can be used or added as training data for machine learning, and the reliance on this work. It excludes verification work for evaluating the degree.

In the present specification, a worker means an individual or an organization that performs crowdsourcing work provided by a verification crowdsourcing work providing device based on actual work and receives a predetermined reward. Projects that include crowdsourcing operations include various types of crowdsourcing operations such as source data collection or data annotation. For example, when the crowdsourcing work is an annotation processing work, the crowdsourcing work providing device for verification based on the production work provides the worker with an image containing a specific object or text, and the work. The person performs an annotation processing operation of designating a range of objects or texts included in the image and inputting what the objects or texts contained in the specified range are. The result of the annotation processing work performed by the worker is transmitted to the automation device for acceptance inspection for the work of performing the data annotation processing using the verification data.

An inspector means an individual or group that inspects the results of crowdsourcing work uploaded by the worker. Depending on the embodiment, the inspector may be understood to include not a human being but a learned model, a program, a terminal, etc. that can perform an automatic inspection.

However, when accepting all crowdsourcing work results, depending on the number of inspectors, a bottleneck phenomenon in the generation of learning data may occur during the acceptance process, and a method for improving this may occur. Development is required.

Therefore, the method and device for providing the crowdsourcing work for verification based on the actual work of the present invention is a method for automatically verifying the crowdsourcing work performed by the worker before the inspector inspects it. Based on the result of the verification crowdsourcing work, it is possible to allow the inspector to inspect only the result of the production crowdsourcing work of the worker who has passed a predetermined standard before proceeding with the acceptance inspection.

Not only that, by configuring the crowdsourcing work for verification based on the crowdsourcing work that passed the acceptance inspection among the results of the crowdsourcing work that has already been performed, the crowdsourcing work for verification can be sustained. By updating, when providing the verification crowdsourcing work to the worker, it is possible to provide a new verification crowdsourcing work that does not overlap and improve the effect on the provision of the verification crowdsourcing work.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

With reference to FIG. 1, the crowdsourcing service is composed of 10 clients, 20 service providers, and 30 masses.

The client 10 means a company or an individual who requests a crowdsourcing project (hereinafter referred to as a project).

The client 10 requests a project for the purpose of collecting source data, data annotation, or the like. Collecting source data means collecting unprocessed data, such as collecting recorded audio, collecting photographs, and so on. Data annotation means inputting relevant annotation data into source data such as text, photos, and videos. For example, data annotation may, but is not limited to, searching for an individual in a given text, searching for similar sentences, and so on. On the other hand, the above-mentioned type of project is only one embodiment, and various projects can be handled in the present invention according to the design of the client.

The service provider 20 means a company that provides a crowdsourcing service.

When the service provider 20 receives a project request for a product or service from the client 10, the service provider 20 assigns the work for the project to the general public 30 and receives the work result from the public 30. After that, the output (final product) extracted based on the work result is provided to the client 10.

Here, the service provider 20 provides the crowdsourcing service to the client 10 and the public 30 via the crowdsourcing platform (hereinafter referred to as the platform). That is, when the service provider 20 receives a project request from the client 10, the service provider 20 opens the project on the platform. After that, when the work result for the opened project is received from the public 30, the project can be completed on the platform, the output can be extracted and provided to the requester 10.

Mass 30 means the general public participating in a project opened on the platform. Here, the public 30 can participate in a project opened on the platform through an application or website provided by the service provider 20.

The mass 30 is composed of a worker 32 and an inspector 34.

Worker 32 decides to participate in a specific project among a plurality of projects opened on the platform. After that, the worker 32 performs work such as collecting source data or data annotation, and sends this to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects opened on the platform. After that, the inspector 34 inspects the work result performed by the worker 32. As a result of the acceptance inspection, the inspector 34 can pass the work or remand the work, and can input the reason for the remand when the work is remanded.

FIG. 2 is a flowchart illustrating a progress process of a crowdsourcing project according to an embodiment of the present invention.

First, the client 10 requests the service provider 20 for one or more projects (S11).

After that, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 can determine the grade in consideration of the difficulty level of the project before opening the project. That is, it is possible to determine which grade or higher the mass 30 should expose (present) the project according to the difficulty level. This will increase the reliability of the work results of the project.

After that, the service provider 20 allocates the work to the workers 32 of the corresponding grade or higher according to the grade of the project (S13).

After that, the worker 32 performs the assigned work (S14). At this time, the worker 32 does not perform the work for the work whose work itself is impossible for some reason, and can input the reason why the work is impossible.

After that, the service provider 20 receives the work result from the worker 32 (S15), and assigns the acceptance inspection work for the work result to the inspector 34 (S16).

After that, the inspector 34 performs the assigned acceptance inspection (S17). At this time, the inspector 34 determines that the work has been performed normally, determines that the work has passed the acceptance inspection, and determines that the work to be inspected is incorrect, and decides to remand the work. When deciding to remand, the inspector 34 inputs the reason for remand for what reason the work is determined to be incorrect.

After that, the service provider 20 receives the acceptance result from the inspector 34 (S18).

If the acceptance result is passed, the service provider 20 uses the corresponding work result as valid data and extracts the output at the end of the project based on this.

If the acceptance result is a remand, the service provider 20 can internally perform the acceptance inspection again or assign the worker 32 to perform the work again.

After that, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the relevant project (S19), and calculates the final result based on the secured valid data. Provided to requester 10 (S20).

At this time, before the end of the project, the service provider 20 evaluates the execution results of the worker 32 and the inspector 34, calculates the work cost and the inspector cost according to the evaluation, and the worker 32 and the inspector 34. Pay to.

On the other hand, in FIGS. 1 and 2, the terms 10 requester, 20 service provider, 32 worker, and 34 inspector are simply expressed, but these are smartphones, tablets, PDAs, laptops, operated by each participant. It means a computer device such as a desktop or a server or a telecommunications device.

FIG. 3 is a schematic sequence diagram of a method of providing a verification crowdsourcing operation based on a production operation according to an embodiment of the present invention.

With reference to FIG. 3, the methods for providing the verification crowdsourcing work based on the production work are the stage of constructing the verification crowdsourcing work pool (S110) and the stage of receiving the production crowdsourcing group work result (S130). And the stage of transmitting one or more verification crowdsourcing work to the worker terminal (S150), the stage of receiving the result of one or more verification crowdsourcing work from the worker terminal (S170), and for production. It includes a stage (S190) of determining whether or not the result of the crowdsourcing group work is valid.

In the stage of constructing the verification crowdsourcing work pool (S110), the verification crowdsourcing work pool is constructed in real time before and / or when the project work is provided to the worker. Including doing.

The specific stage of constructing the crowdsourcing work pool for verification (S110) will be described later in the description of FIGS. 5 to 8.

In the production crowdsourcing group work result reception stage (S130), the result of the production crowdsourcing group work is received from the worker terminal, and the production crowdsourcing group work is composed of one or more production crowdsourcing work. NS. The production crowdsourcing work consists of the work requested by the customer who requested the crowdsourcing work.

Here, the crowdsourcing work includes various types of crowdsourcing work such as collection of source data or data annotation as described above. An example of crowdsourcing work is annotation work processing work, which will be described later in the description of FIG.

The customer requests the administrator of the method and device for providing the verification crowdsourcing work based on the production work to perform the predetermined crowdsourcing work. The manager requests one or more workers to perform a predetermined crowdsourcing work that has been requested, and pays a reward according to the work result of the workers.

A production crowdsourcing group operation composed of one or more production crowdsourcing operations will be described with reference to FIG.

At the stage of transmitting one or more verification crowdsourcing operations to the worker terminal (S150), one or more verification crowdsourcing operations corresponding to the production crowdsourcing group work are selected in the verification crowdsourcing work pool. And send it to the worker terminal.

The verification crowdsourcing work pool contains one or more verification crowdsourcing operations, and the verification crowdsourcing work is selected and sent to the worker terminal based on a certain standard or at random. To further explain, one or more verification crowdsourcing operations are provided to the worker terminal for each production crowdsourcing group operation, and one or more verification crowdsourcing operations provided at this time are provided. , Selected from within the crowdsourcing work pool for verification.

Since the answer to each verification crowdsourcing operation is predetermined, the computer must receive the predetermined answer from the worker terminal.

Is the stage (S150) of transmitting one or more verification crowdsourcing operations to the worker terminal one or more verification crowdsourcing operations generated from the results of the production crowdsourcing operations received from the same worker? It is possible to send one or more verification crowdsourcing operations in consideration of whether or not it is one or more verification crowdsourcing operations already provided to the same worker.

As an embodiment, the step of transmitting one or more verification crowdsourcing operations to the worker terminal (S150) excludes the verification crowdsourcing work already provided to the same worker terminal as the worker terminal. Send one or more verification crowdsourcing operations to the worker terminal. This is because, in the case of the verification crowdsourcing work already provided to the same worker terminal, the worker knows the answer in advance and the verification effect is reduced.

As another example, a specific verification crowdsourcing work in the verification crowdsourcing work pool is generated based on the result of one or more production crowdsourcing work received from the same worker terminal as the worker terminal. If so, at the stage of transmitting one or more verification crowdsourcing work to the worker terminal (S150), the worker performs one or more verification crowdsourcing work excluding the specific verification crowdsourcing work. Send to the terminal. Even in the case of the verification crowdsourcing work generated based on the result of the production crowdsourcing work received from the same worker terminal, the verification effect is reduced because the worker knows the answer in advance. Applicable verification crowdsourcing work must also be excluded.

At the stage (S150) of transmitting one or more verification crowdsourcing operations to the worker terminal, the computer can read one or more verification crowdsourcing operations processed in the verification crowdsourcing operation pool. Here, processing can mean generating a dataset using the answers of one or more crowdsourcing operations and one or more crowdsourcing operations.

The computer selects one or more verification crowdsourcing tasks from the verification crowdsourcing task pool. The computer selects from the verification crowdsourcing work pool one or more verification crowdsourcing work problems that correspond to the production crowdsourcing work. Therefore, the verification crowdsourcing work and the production crowdsourcing work can be performed by the work having the same category. In addition, the computer can select one or more of the production crowdsourcing operations as the verification crowdsourcing operation. Workers can themselves during crowdsourcing work by putting the verification crowdsourcing work and the production crowdsourcing work in the same category, or by selecting one or more of the production crowdsourcing work as the verification crowdsourcing work. Can't know whether they are doing the production crowdsourcing work or the verification crowdsourcing work, so they will be doing the crowdsourcing work. In addition, the result of the verification crowdsourcing work can represent the result of the production crowdsourcing group work.

One or more production crowdsourcing operations are stored and managed in a verification crowdsourcing work pool, and one or more verification crowdsourcing operations are stored in one or more production crowdsourcing work pools. Selected from crowdsourcing work. The number of one or more verification crowdsourcing operations can also be determined corresponding to the number of the one or more production crowdsourcing operations stored in the verification crowdsourcing work pool.

The verification crowdsourcing work generated in the present invention is as described later in FIGS. 6 to 8, and the verification crowdsourcing work prepared in advance before generating the verification crowdsourcing work acquired from the production work is a customer. Selected from the entire production crowdsourcing work requested by. The worker performs a part of the whole verification crowdsourcing work during the production crowdsourcing work, and the result of the production crowdsourcing work performed by the worker is the result of the whole verification crowdsourcing work. It is decided whether or not to do the other part of.

The database stores and manages one or more production crowdsourcing operations received from the customer's terminal, stores and manages verification crowdsourcing operations and answers selected from the database control unit (not shown), and works. Stores and manages the answers to one or more production crowdsourcing operations received from a person's terminal.

At the stage of receiving the results of one or more verification crowdsourcing operations from the worker terminal (S170), the results of one or more verification crowdsourcing operations corresponding to the production crowdsourcing group work are received from the worker terminal. do.

The predetermined answers corresponding to the verification crowdsourcing work and the verification crowdsourcing work consist of a data set of the verification crowdsourcing work. The verification crowdsourcing work can be selected from the production crowdsourcing work, which is a predetermined crowdsourcing work requested by the customer. As an example, the predetermined answer corresponding to the verification crowdsourcing work can be obtained in advance by the administrator of the verification device of the production crowdsourcing work using the verification crowdsourcing work for the verification crowdsourcing work. It can be the answer you created. The administrator can divide a predetermined crowdsourcing work requested by a customer into a predetermined field and configure a data set of crowdsourcing work and an answer for each field.

However, for the verification crowdsourcing work and the corresponding predetermined answer, the selection and answer of the verification crowdsourcing work were generated before the administrator requested the worker to carry out the project. Even so, the amount of work that must be done in one project is very large, and it may be necessary to provide verification crowdsourcing work more frequently to workers who are often sent back, so in advance There is a risk that the number of verification crowdsourcing operations that have been generated will be insufficient.

If all the crowdsourcing work for verification generated in advance is provided, and if the crowdsourcing work for verification that is duplicated and generated in advance is provided again, the worker can memorize the answer of the work and the crowd for verification. Since it is easy to respond to sourcing work, it is possible to draw unsolvable conclusions for the essential issues that provide verification crowdsourcing work.

Therefore, the present invention makes it possible to continuously provide a new verification crowdsourcing work even if all the verification crowdsourcing work generated in advance is provided, and one or more of the present inventions. The verification crowdsourcing work corresponds to one or more in the verification crowdsourcing work pool composed of one or more acceptance-passing crowdsourcing work.

At this time, the crowdsourcing work that has passed the acceptance inspection is the work that has passed the acceptance inspection from the inspector among the one or more crowdsourcing work that has already been performed, and the one or more crowdsourcing work that has already been performed is the production crowd. This is the work done before the sourcing group work.

The verification crowdsourcing work acquired based on the production work will be described in detail with reference to FIGS. 7 and 8.

At the stage of determining whether or not the result of the production crowdsourcing group work is valid (S190), whether or not the result of the production crowdsourcing group work is valid based on the result of one or more verification crowdsourcing work. To judge.

The computer determines whether the results of one or more production crowdsourcing operations are classified as valid data or ineffective data.

If the result of the verification crowdsourcing work matches a predetermined answer, the computer processes the result of the production crowdsourcing group work as valid, and the result of the verification crowdsourcing work is determined in advance. If it does not match the answer, the result of the production crowdsourcing group work is treated as invalid. Therefore, the production crowdsourcing work defined as a result of the ineffective crowdsourcing work is not accepted by the inspector, so that the amount of acceptance by the inspector can be reduced. In addition, the production crowdsourcing group work defined as a result of effective crowdsourcing work is inspected by the inspector.

That is, although not particularly shown, the method of providing the verification crowdsourcing work based on the production work of the present invention is the result of the stage of constructing the verification crowdsourcing work pool (S110) or the production crowdsourcing group work. In the stage of determining whether or not is valid (S190), a step of deciding whether or not to perform acceptance inspection for the result of the production crowdsourcing group work is further included depending on whether or not the result of the production crowdsourcing group work is valid. be able to.

At the stage of deciding whether or not to perform acceptance inspection, if the result of the verification crowdsourcing work matches the predetermined answer, it is decided to perform the acceptance inspection on the result of the production crowdsourcing group work. If the result of the verification crowdsourcing work does not match the predetermined answer, the result of the corresponding production crowdsourcing group work is deleted and it is decided not to perform the acceptance inspection.

FIG. 4 is a schematic illustration of annotation processing for an image containing an object.

With reference to FIG. 4, an example in which an annotation process for an image including a bed is performed as a crowdsourcing operation is shown. The production crowdsourcing work and the verification crowdsourcing work can be in the form shown in FIG.

The image 200 shown in FIG. 4 includes at least one object. The operator can select a specific area 210 of the image 200 and input or select information 220 that describes the object contained in the selected area 210. For example, one image 200 can include a plurality of objects, and the worker can perform crowdsourcing work of performing annotation processing on one or more objects included in the image 200.

In the embodiment shown in FIG. 4, the result of the crowdsourcing work as the annotation processing work is the coordinates of the specific area 210 and the information about the object contained in the specific area 210 (for example, in the specific area 210). Information that the included object is a bed) can be included.

5 to 8 are specific explanations of the stage (S110) of constructing the crowdsourcing work pool for verification.

The verification crowdsourcing work pool at the stage of constructing the verification crowdsourcing work pool (S110) is composed of the production work platform verification crowdsourcing work derived based on the result of the production crowdsourcing work as an example. As another embodiment, the verification crowdsourcing work of the above embodiment is further configured to include the verification crowdsourcing work generated in advance by the administrator before the execution of the crowdsourcing work project. It is possible.

If you want to build a crowdsourcing work pool for verification that includes the crowdsourcing work for verification generated in advance by the administrator before executing the crowdsourcing work project, select from the crowdsourcing work pool for verification and work. In the stage of transmitting to the user terminal (S150), after the pre-generated verification crowdsourcing work is exhausted (all used), the verification cloud sourcing work of the production work platform is transmitted to the worker terminal.

Therefore, in the initial progress stage of the crowdsourcing work project, the crowdsourcing work for verification generated in advance is provided as the crowdsourcing work for verification, and when the crowdsourcing work for verification generated in advance is exhausted, the production work platform is verified. By providing crowdsourcing work for workers, from the initial stage to the final stage of work, we continuously provide new verification crowdsourcing work to workers and verify the results of crowdsourcing work for workers. can do.

FIG. 5 is a diagram illustrating a method of constructing a crowdsourcing work pool for verification.

With reference to FIG. 5, the method of constructing the verification crowdsourcing work pool is the stage of extracting the result of the acceptance inspection passing crowdsourcing work (S111) and the stage of generating the verification crowdsourcing work of the production work platform (S113). ) And the stage (S115) of including the crowdsourcing work for verification of the production work platform in the crowdsourcing work pool for verification.

Passing acceptance The stage (S111) of extracting the results of crowdsourcing work passes the acceptance of the inspector from the results of one or more production crowdsourcing work that has already been performed in real time while the crowdsourcing work project is in progress. Extract the results of crowdsourcing work that has passed the acceptance inspection.

The stage of generating the crowdsourcing work for verification of the production work platform (S113) is based on the result of the acceptance-passing crowdsourcing work that satisfies the predetermined conditions among the results of the acceptance-passing crowdsourcing work extracted in the S111 stage. , Production work platform Generate crowdsourcing work.

Here, the predetermined condition corresponds to a predetermined order based on the time required for acceptance inspection.

Generating the crowdsourcing work of the production work platform based on the result of the crowdsourcing work that passed the acceptance inspection is, as an example, generating the crowdsourcing work of the production work platform based on the result of the crowdsourcing work that passed the acceptance inspection. It is possible to request another inspector to re-inspect the result of the crowdsourcing work that has passed the acceptance inspection, and to generate the crowdsourcing work of the production work platform according to the result of the re-inspection.

At this time, when the crowdsourcing work for verification of the production work infrastructure is generated when the other inspectors are re-inspected, the crowdsourcing work that satisfies the predetermined conditions is satisfied from the result of the extracted crowdsourcing work. Regarding the stage of extracting the results, the stage of requesting one or more inspectors different from the inspector who performed the relevant inspection for the result of the crowdsourcing work that passed the acceptance inspection that meets the predetermined conditions, and the stage of requesting re-inspection, and the result of the re-inspection, It can include a stage of generating a crowdsourcing work for verification of a production work platform based on the result of a re-inspection passing crowdsourcing work that has passed the acceptance inspection by one or more or one or more inspectors.

In the stage (S115) of including the crowdsourcing work for verification of the production work platform in the crowdsourcing work pool for verification, the crowdsourcing work for verification of the production work platform generated in the S113 stage is included in the crowdsourcing work pool for verification.

FIG. 6 is a diagram illustrating a method of providing a production crowdsourcing work and a verification crowdsourcing work.

With reference to FIG. 6, since the arrows indicate the order of provision, it can be confirmed that the verification crowdsourcing work 60 is provided after the production crowdsourcing group work 50 is provided to the workers.

Production crowdsourcing group work 50 includes one or more production crowdsourcing operations, and in FIG. 6, six production crowdsourcing operations 51, 52, 53, 54, 55, 56 are exemplified. showed that. However, since the production crowdsourcing group work may include one or more production crowdsourcing work, it may be less or more than in FIG.

Further, in FIG. 6, only one unit in which the verification crowdsourcing work 60 is provided is shown, but in FIG. 6 until the work of the production crowdsourcing work 51 to 56 of the corresponding project is completed and the acceptance is passed. Providing work can be provided iteratively. That is, after the production crowdsourcing group work 50 is provided, the verification crowdsourcing work 60 can be provided, and after the production crowdsourcing group work 50 is provided, the verification crowdsourcing work 60 and the like can be repeatedly provided.

7 and 8 are illustrations illustrating a method of acquiring the verification crowdsourcing work from the result of the production crowdsourcing work.

Since the project of the present invention is not performed by one worker but by a plurality of workers as crowdsourcing, the production already performed before the production crowdsourcing work is performed by a specific worker. Crowdsourcing operations 71, 72, 73, 74, 75, 76 may exist. When a specific worker first performs the production crowdsourcing work, the production crowdsourcing work 71 to 76 that has already been performed does not exist. In such a case, as described above, the administrator It is possible to provide a crowdsourcing work for verification generated in advance. However, in FIGS. 7 and 8, it will be described that the verification crowdsourcing work is generated based on the production work corresponding to the case where the production crowdsourcing work 71 to 76 that has already been performed exists.

The verification crowdsourcing work pool can be updated in real time, and when the acceptance-passing crowdsourcing work 81-84 is extracted in real time, the extracted crowdsourcing work 81-84 is used for verification crowdsourcing work. You can update the crowdsourcing work pool for verification by adding as.

Here, the extracted acceptance-passing crowdsourcing work can include, as an example, simply the acceptance-passing crowdsourcing operations 81 to 84 that have passed the acceptance inspection, and as another example, FIG. 7 described later. As described above, the acceptance-passing crowdsourcing operations 81 to 83 extracted in a predetermined order can be included, and as yet another embodiment, as described in FIG. 8 described later, among the predetermined orders. The crowdsourcing work 81 that has passed the acceptance of other inspectors can be included.

In addition, the verification crowdsourcing work pool can further include the newly extracted crowdsourcing work that has passed the acceptance inspection, excluding the verification crowdsourcing work used by the workers.

As an example, if the verification crowdsourcing work pool is different for each worker, only the number of newly extracted acceptance-passing crowdsourcing that has already been provided to the worker is excluded. Work is further included in the verification crowdsourcing work pool.

The above embodiment is based on the case where the number of verification crowdsourcing operations included in the verification crowdsourcing work pool is specified (for example, the number of verification crowdsourcing operations is maintained at 100). It may be applicable.

As another embodiment, the verification crowdsourcing work pool can be built cumulatively, and when the verification crowdsourcing work pool is the same for each worker, it is the same every time the verification crowdsourcing work is provided. Provided except for the work done by the worker.

The above embodiment corresponds to the case where the number of verification crowdsourcing operations is not specified, and is excluded when the number of verification crowdsourcing operations is not specified. Every time the verification crowdsourcing work 60 is extracted, the extracted verification crowdsourcing work 60 can be continuously included in the verification crowdsourcing work pool without considering the sourcing work.

The number of the production crowdsourcing operations 71 to 76, the acceptance-passing crowdsourcing operations 80, and the acceptance-passing crowdsourcing operations 81 to 84 shown in FIG. 7 is arbitrary, and if it is one or more. , Any number can be included.

Referring to FIG. 7, the production crowdsourcing operations 71 to 76 already performed are the production crowdsourcing operations already performed by the worker, and correspond to the operations before the acceptance inspection of the inspector is completed. When the acceptance inspection for the production crowdsourcing operations 71 to 76 that have already been performed is completed, the production crowdsourcing operation 73, which has passed the acceptance inspection, that is, the work by the worker and then the acceptance inspection by the inspector. 74, 75, 76 are classified into acceptance acceptance passing crowdsourcing work 80.

The computer arranges the classified crowdsourcing operations 80 that have passed the acceptance inspection in the order of the longest acceptance time, as shown in the example of FIG. 7, and the order is determined in the order of the longest acceptance time. Only the acceptance-passing crowdsourcing operations 81 to 83 corresponding to the range (for example, from the 1st place to the nth place) are extracted.

The long acceptance time means that the work must be done more carefully than other work, so if you extract in the order of the longest acceptance time, it is in line with the purpose of providing crowdsourcing work for verification. It is more effective because it does not require a separate acceptance inspection time for defective work.

In addition, the production crowdsourcing operations 71-76 and the acceptance-passing crowdsourcing operations 81-84 that have already been performed are continuously generated in real time, but the criteria for extracting up to a predetermined order are those of the administrator or the computer. It can be preset by setting, and can be extracted in units of pieces or hours.

The acceptance-passing crowdsourcing operations 81 to 83 extracted by a predetermined order according to FIG. 7 can be immediately included in the verification crowdsourcing operation pool.

However, the inspector may also have an error in the acceptance, such as not being able to concentrate on the work, so in order to prevent this and provide a more verified crowdsourcing work 60 for verification, verification through additional inspector acceptance. Crowdsourcing work 60 can be extracted as the final verification crowdsourcing work 60 included in the crowdsourcing work pool.

Referring to FIG. 8, a method of extracting the verification crowdsourcing work 60 through the acceptance inspection passing crowdsourcing operations 81 to 83 extracted up to a predetermined order in FIG. 7 through additional acceptance inspection work of another inspector. You can check.

First, assuming that the inspector who inspected in FIG. 7 is the first inspector, in FIG. 8, the second inspector 34-1 and the third inspector 34-2 are additionally inspected. At this time, it is shown in FIG. 8 that only two inspectors will be additionally inspected, but the number of inspectors who additionally inspect is one or more, and whether it is predetermined or the administrator It can be set or determined based on certain criteria set.

More specifically, FIG. 8 shows that in FIG. 7, the acceptance inspection by the first inspector is passed, and the acceptance inspection passing crowdsourcing operations 81 to 83 extracted in a predetermined order are performed by the second inspector 34-1 and The third inspector 34-2 will be additionally inspected.

The second inspector 34-1 passed the acceptance inspection only for the acceptance-passing crowdsourcing work of 81 and 82, and did not pass it for the acceptance-passing crowdsourcing work of 83.

The third inspector 34-2 passed the acceptance inspection only for the crowdsourcing work passing the acceptance inspection of 81, and did not pass the acceptance inspection for the crowdsourcing work passing the acceptance inspection of 82 and 83.

Therefore, the work that has passed all the acceptances of the second inspector 34-1 and the third inspector 34-2, which are additional inspectors, is the 81 inspector passing crowdsourcing work, and only the corresponding inspector passing crowdsourcing work. Is the verification crowdsourcing work 60, which will be included in the verification crowdsourcing work pool.

Although not particularly shown, a method of providing a verification crowdsourcing work based on the production work of the present invention is added after the stage (S190) of determining whether or not the result of the production crowdsourcing group work is valid. It can further include the stage of receiving the results of a typical production crowdsourcing group work and the stage of receiving the results of additional verification crowdsourcing work.

1) When the result of n production crowdsourcing work is not valid The computer immediately after defining the result of n production crowdsourcing work as invalid crowdsourcing work, n-i is sent to the worker's terminal. Send additional production crowdsourcing operations (where i is a natural number greater than or equal to 1).

The computer receives the results of ni additional production crowdsourcing operations from the worker's terminal. It then loads one or more pre-prepared additional verification crowdsourcing tasks that correspond to n-i additional production crowdsourcing tasks.

Immediately after receiving the results of ni additional production crowdsourcing operations, the computer transmits the one or more additional verification crowdsourcing operations to the worker's terminal. Then, the result of one or more additional verification crowdsourcing operations is received from the worker's terminal.

If the worker cannot successfully perform the verification crowdsourcing work, then the verification crowdsourcing work will be transmitted and received faster, and then the order in which the verification crowdsourcing work is received depends on i. It is determined and i is modified accordingly to the number of tasks defined as ineffective crowdsourcing tasks. For example, i can be the number of previously ineffective crowdsourcing operations defined as the number of operations divided by a predetermined natural number greater than or equal to one. Alternatively, i can be a given number of natural numbers greater than or equal to one that corresponds to the number of tasks previously defined as ineffective crowdsourcing tasks.

When i is less than or equal to the predetermined standard, the worker who performed the crowdsourcing work for production and the crowdsourcing work for verification is defined and classified as an unreliable worker, and one computer is used for the terminal of the worker. Do not send the above production crowdsourcing work. For example, when i is 2 or less, the worker cannot perform the production crowdsourcing work any more. Defining a worker as an unreliable worker means that he / she is unreliable for one particular crowdsourcing task, and when a new or existing customer requests an administrator for a new crowdsourcing task. Workers who have been classified as unreliable workers in the previous crowdsourcing work can also perform new production crowdsourcing work.

The customer request receiving unit (not shown) can receive the reliability of the crowdsourcing work requested from the customer's terminal. i may be a value changed according to the reliability of the crowdsourcing work requested from the customer terminal. i can be the number of tasks defined as ineffective crowdsourcing divided by a predetermined number. For example, i can be the number of ineffective crowdsourcing operations divided by two. The predetermined number can be determined by the reliability of the crowdsourcing work requested by the customer. For example, if the reliability of the crowdsourcing work requested by the customer is high, the predetermined number can be reduced to a low number. As a result, i becomes a relatively high number, and the time point at which the verification crowdsourcing work is transmitted is relatively advanced.

2) When the result of n production crowdsourcing work is valid The computer defines the result of n production crowdsourcing work as a valid crowdsourcing work, and immediately after that, n + j (n + j) ( However, j transmits an additional production crowdsourcing work (a natural number of 1 or more).

The computer receives the results of n + j additional production crowdsourcing operations from the worker's terminal.

Then, one or more pre-prepared additional verification crowdsourcing operations are loaded, corresponding to n + j additional production crowdsourcing operations.

Immediately after receiving the results of n + j additional production crowdsourcing operations, the computer sends one or more additional verification crowdsourcing operations to the worker's terminal. Then, the result of one or more additional verification crowdsourcing operations is received from the worker's terminal.

If the worker normally performs the verification crowdsourcing work, the verification crowdsourcing work will be transmitted and received later, and the order of receiving the verification crowdsourcing work after that depends on j. It is determined and j is modified according to the number of tasks defined as valid crowdsourcing tasks. For example, j can be the number of previously defined crowdsourced operations divided by a predetermined natural number greater than or equal to 1. Alternatively, j can be a given number of natural numbers greater than or equal to one that corresponds to the number of tasks previously defined as valid crowdsourcing tasks.

When j is equal to or higher than a predetermined standard, the worker who has performed the relevant production crowdsourcing work and the verification crowdsourcing work is defined and classified as a reliable worker, and the computer is set to the terminal of the corresponding worker. Do not send more than one verification crowdsourcing operation. Defining a worker as a trusted worker means being reliable for one particular crowdsourcing task, when a new or existing customer requests an administrator for a new crowdsourcing task. Workers who have been classified as reliable workers in the previous crowdsourcing work may also have to perform new verification crowdsourcing work.

When a worker is classified and defined as a trusted worker, the stage of receiving the results of one or more additional verification crowdsourcing operations that correspond to one or more additional production crowdsourcing operations. Does not progress. Therefore, the result of additional production crowdsourcing work by a reliable worker is defined as an effective crowdsourcing work without the process of verifying with the verification crowdsourcing work.

The customer request receiver (not shown) receives the reliability of the crowdsourcing work requested from the customer's terminal, and j is a value changed according to the reliability of the crowdsourcing work requested by the customer. Can be. j can be a value obtained by dividing the number of effective crowdsourcing operations by a predetermined number. For example, j can be the number of effective crowdsourcing operations divided by two. The predetermined number may be determined by the reliability of the crowdsourcing work requested by the customer. For example, if the reliability of the crowdsourcing work requested by the customer is high, the predetermined number can be increased. As a result, j becomes a relatively low number, and the time point at which the verification crowdsourcing work is transmitted (transmitted / received) is relatively advanced.

FIG. 9 is an example diagram for providing a production crowdsourcing work and a verification crowdsourcing work later when the result of the production crowdsourcing work is not valid.

With reference to FIG. 9, if the result of the verification crowdsourcing work does not match the predetermined answer, the result of the production crowdsourcing work is defined and classified as invalid.

If the worker cannot normally perform the verification crowdsourcing work # 1 (310), the result of the n production crowdsourcing work performed before the verification crowdsourcing work # 1 (310) is , Defined and classified as ineffective crowdsourcing work. Then, the worker performs the production crowdsourcing work n-i times, which is lower than the previous n times, and performs the verification crowdsourcing work # 2 (320). Similarly, if the verification crowdsourcing work # 2 (320) cannot be performed normally, it is performed after the verification crowdsourcing work # 1 (310) and before the verification crowdsourcing work # 2 (320). The results of the ni production crowdsourcing operations performed in 1 are defined and classified as ineffective crowdsourcing operations.

Therefore, if the worker does not normally perform the verification crowdsourcing work, the result of the previous production crowdsourcing work is defined and classified as an ineffective crowdsourcing work. Then, the reliability of the worker's ability is evaluated lower than before, and the worker receives the verification crowdsourcing work even faster than before. The verification crowdsourcing work may be transmitted and received relatively quickly, and may be received relatively late, depending on the results of the previous verification crowdsourcing work. Then, when i becomes a predetermined value or less, the corresponding worker becomes an unreliable worker, and thereafter, the verification crowdsourcing work and the production crowdsourcing work cannot be performed.

FIG. 10 is an example diagram for providing the production crowdsourcing work and the verification crowdsourcing work after the result of the production crowdsourcing work is valid.

With reference to FIG. 10, if the result of the verification crowdsourcing work matches a predetermined answer, the result of the production crowdsourcing work is defined and classified as valid.

When the worker normally performs the verification crowdsourcing work # 1 (410), the result of the n production crowdsourcing work performed before the verification crowdsourcing work # 1 (410) is valid. Defined and classified as crowdsourcing work. Then, the worker performs the production crowdsourcing work n + j times, which is more than the previous n times, and performs the verification crowdsourcing work # 2 (420). Similarly, if the verification crowdsourcing work # 2 (420) is normally performed, it is performed after the verification crowdsourcing work # 1 (410) and before the verification crowdsourcing work # 2 (420). The results of n + j production crowdsourcing operations are defined and classified as effective crowdsourcing operations.

Therefore, when the worker normally performs the verification crowdsourcing work, the result of the previous production crowdsourcing work is defined and classified as a valid crowdsourcing work. Then, the reliability of the ability of the worker is evaluated higher than before, and the worker receives the verification crowdsourcing work even later than before. The verification crowdsourcing work may be transmitted and received relatively quickly, and may be transmitted and received relatively slowly, depending on the result of the previous verification crowdsourcing work. Then, when j becomes a predetermined value or more, the corresponding worker becomes a reliable worker, and the result of the production crowdsourcing work is defined as an effective crowdsourcing work without performing the verification crowdsourcing work thereafter. do.

FIG. 11 is a schematic configuration diagram of a verification crowdsourcing work providing device based on the production work according to another embodiment of the present invention.

Referring to FIG. 11, the verification crowdsourcing work providing device based on the production work includes the verification crowdsourcing work pool construction unit 510, the production crowdsourcing group work result receiving unit 520, and the verification crowdsourcing work transmitting unit 530. , The crowdsourcing work result receiving unit 540 for verification and the valid / invalidity determining unit 550 are included.

The components of the crowdsourcing work providing device for verification based on the production work of FIG. 11 correspond to the components of the method described with reference to FIGS. 1 to 10. Hereinafter, duplicate description will be omitted in explaining the function or operation of each component of the device.

The verification crowdsourcing work pool construction unit 510 constructs a verification crowdsourcing work pool composed of verification crowdsourcing work.

The production crowdsourcing group work result receiving unit 520 receives the result of the production crowdsourcing group work from the worker terminal.

The verification crowdsourcing work transmission unit 530 selects one or more verification crowdsourcing work corresponding to the production crowdsourcing group work from the verification crowdsourcing work pool and transmits it to the worker terminal.

The verification crowdsourcing work result receiving unit 540 receives the results of one or more verification crowdsourcing work from the worker terminal.

The valid / invalid determination unit 550 determines whether or not the result of the production crowdsourcing group work is valid based on the result of one or more verification crowdsourcing work.

The steps of methods or algorithms described in connection with the embodiments of the present invention can be realized directly in hardware, in software modules executed by hardware, or by combining them. .. The software modules include RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Program ROM), EPROM (Electrically Eraseble Program Alternatively, it may be present in any type of computer-readable recording medium well known in the art to which the present invention belongs.

Although the embodiments of the present invention have been described above with reference to the accompanying drawings, ordinary engineers in the technical field to which the present invention belongs can use the present invention without changing its technical idea or essential features. It can be understood that it can be carried out in a concrete form of. Therefore, the examples described above should be understood as exemplary in all respects and not restrictive.

10 Client 20 Service provider 30 Mass 32 Worker 34 Inspector 50 Production crowdsourcing group work 60 Verification crowdsourcing work 71, 72, 73, 74, 75, 76 Processed production crowdsourcing work 80, 81 , 82, 83, 84 Crowdsourcing work for acceptance 510 Verification crowdsourcing work Pool construction unit 520 Production crowdsourcing group work result receiving unit 530 Verification crowdsourcing work Transmission unit 540 Verification crowdsourcing work result receiving unit 550 Validity Judgment department

Claims (8)

It ’s a computer-based method,
At the stage of building a crowdsourcing work pool for verification,
A production crowdsourcing group that receives the results of production crowdsourcing group work from a worker terminal, and the production crowdsourcing group work is composed of one or more production crowdsourcing work. Work result reception stage and
A stage in which one or more verification crowdsourcing operations corresponding to the production crowdsourcing group work are selected from the verification crowdsourcing work pool and transmitted to the worker terminal.
At the stage of receiving the result of one or more verification crowdsourcing operations from the worker terminal, and
Including the stage of determining whether or not the result of the production crowdsourcing group work is valid based on the result of the one or more verification crowdsourcing work.
The verification crowdsourcing work pool includes a plurality of verification crowdsourcing operations.
The stage of building the crowdsourcing work pool for verification is
Crowdsourcing work The stage of extracting the results of the crowdsourcing work that has passed the acceptance of the inspector from the results of one or more production crowdsourcing work that has already been performed in real time while the project is in progress.
Among the extracted results of the crowdsourcing work that passed the acceptance inspection, the stage of generating the crowdsourcing work for verification of the production work platform based on the result of the crowdsourcing work that passed the acceptance inspection that satisfies the predetermined conditions, and
Including the stage of including the crowdsourcing work for verification of the generated production work platform in the crowdsourcing work pool for verification.
The predetermined conditions are
It corresponds to a predetermined ranking based on the time taken for acceptance inspection.
The stage of generating the verification crowdsourcing work is
From the extracted results of the acceptance-passing crowdsourcing work, the stage of extracting the results of the acceptance-passing crowdsourcing work that satisfies the predetermined conditions, and
Regarding the result of crowdsourcing work that passes the acceptance inspection that meets the predetermined conditions, the stage of requesting re-inspection from one or more inspectors different from the inspector, and
As a result of the re-acceptance, the stage of generating the crowdsourcing work for the production work platform verification based on the result of the re-acceptance passing crowdsourcing work that has passed the acceptance inspection by all of the one or more inspectors, and
How to provide crowdsourcing work for verification based on production work, including. At the stage of transmitting one or more verification crowdsourcing operations to the worker terminal,
The first aspect of claim 1 is characterized in that one or more verification crowdsourcing operations are transmitted to the worker terminal, excluding the verification crowdsourcing work already provided to the same worker terminal as the worker terminal. A method of providing a crowdsourcing operation for verification based on the production operation described. When a specific verification crowdsourcing operation in the verification crowdsourcing work pool is generated based on the result of one or more production crowdsourcing operations received from the same worker terminal as the worker terminal. for,
The stage of transmitting one or more verification crowdsourcing operations to the worker terminal is
The verification crowdsourcing work based on the production work according to claim 1, wherein one or more verification crowdsourcing work excluding the specific verification crowdsourcing work is transmitted to the worker terminal. How to provide. The stage of building the crowdsourcing work pool for verification is
The production work according to claim 1, further comprising including the verification crowdsourcing work generated in advance by the administrator in the verification crowdsourcing work pool before the execution of the crowdsourcing work project. How to provide crowdsourcing work for verification based on. The stage of selecting from the verification crowdsourcing work pool and transmitting to the worker terminal is
The verification based on the production work according to claim 4, wherein the crowdsourcing work for verification of the production work platform is transmitted to the worker terminal after the crowdsourcing work for verification generated in advance is exhausted. How to provide crowdsourcing work for. Further including a step of deciding whether or not to inspect the result of the production crowdsourcing group work depending on whether the result of the production crowdsourcing group work is valid or not.
The stage of deciding whether or not to accept the results of the production crowdsourcing group work is
If the result of the verification crowdsourcing work matches the predetermined answer, it is decided to inspect the result of the corresponding production crowdsourcing group work.
When the result of the verification crowdsourcing work does not match the predetermined answer, the claim is characterized in that the result of the corresponding production crowdsourcing group work is deleted and it is decided not to perform acceptance inspection. A method for providing a verification crowdsourcing work based on the production work according to item 1. A crowdsourcing work-providing computer program for verification based on production work, which is combined with a computer as hardware and stored in a medium to execute the method according to any one of claims 1 to 6. Building a crowdsourcing work pool for verification With the crowdsourcing work pool construction department for verification,
A production crowdsourcing group that receives the results of production crowdsourcing group work from a worker terminal, and the production crowdsourcing group work is composed of one or more production crowdsourcing work. Work result receiver and
A verification crowdsourcing work transmission unit that selects one or more verification crowdsourcing operations corresponding to the production crowdsourcing group work from the verification crowdsourcing work pool and transmits them to the worker terminal.
A verification crowdsourcing work result receiving unit that receives the results of one or more verification crowdsourcing operations from the worker terminal, and
A valid / invalid judgment unit for determining whether or not the result of the production crowdsourcing group work is valid based on the result of the one or more verification crowdsourcing work is included.
The verification crowdsourcing work pool includes a plurality of verification crowdsourcing operations.
The verification crowdsourcing work pool construction unit
Crowdsourcing work In real time while the project is in progress, the results of one or more production crowdsourcing work that has already been performed are extracted from the results of the crowdsourcing work that has passed the acceptance of the inspector.
Among the results of the extracted crowdsourcing work that passed the acceptance inspection, based on the result of the crowdsourcing work that passed the acceptance inspection that satisfies the predetermined conditions, a crowdsourcing work for verification of the production work platform was generated.
The crowdsourcing work for verification of the generated production work platform is included in the crowdsourcing work pool for verification.
The predetermined conditions are
It corresponds to a predetermined ranking based on the time taken for acceptance inspection.
The verification crowdsourcing work pool construction unit
From the extracted results of the acceptance-passing crowdsourcing work, the results of the acceptance-passing crowdsourcing work that satisfy the predetermined conditions are extracted.
Regarding the results of the acceptance-passing crowdsourcing work that meets the predetermined conditions, request re-inspection from one or more inspectors different from the inspectors.
As a result of the re-acceptance, a crowdsourcing work for verification of the production work platform is generated based on the result of the re-acceptance passing crowdsourcing work that has passed the acceptance inspection by all of the one or more inspectors. Sourcing work providing device.

Images