JP7084817B2 - Subjective evaluation device and program - Google Patents

Description

The present invention relates to a subjective evaluation device and a program that presents random evaluation targets in the order of arrangement.

When subjectively evaluating multiple evaluation targets, the order of the evaluation targets in the trial for each evaluator needs to be sorted by evaluator using some randomization method for the purpose of eliminating the order effect (for example,). See Non-Patent Documents 3 and 4). It is also recommended to randomize the order of evaluation targets in the literature describing the actual subjective evaluation method (see, for example, Non-Patent Documents 1, 2 and 5).

Recommendation ITU-R BS.1116-3, 2015 Recommendation ITU-R BS.1534-3, 2015 Togerson et al., "Randomized Controlled Trial Design", Nihon Hyoronsha, 2010 Tango, "Randomized Controlled Trials", Asakura Shoten, 2003 Watanabe, "Evaluation Method for Deterioration of Audio Signals", 2007, Journal of Acoustical Society of Japan, Vol. 63, pp.686-692

Conventionally, the randomization procedure in the subjective evaluation device often applies the randomization function of the computer, such as using the land function attached to Excel (registered trademark). However, these randomization algorithms are mechanisms that use pseudo-random numbers, and do not guarantee the randomness of the partial order. For example, although the sequences [1 2 3 4 5] and [3 4 5 1 2] are different, the order [1 2] and [3 4 5] are common to both. Therefore, it is necessary to visually check the sort order output by the computer and check whether there is any bias in the sort order. However, the evaluation of the degree of randomness and the exclusion of a specific order by the experimenter may be subject to a specific bias, and may not be a fair experimental design. In addition, when the number of evaluation targets and the number of trials are large, it is practically impossible for the experimenter to manage all the order, and there is a limit to the randomization by these methods.

An object of the present invention made in view of such circumstances is a subjective evaluation device capable of automatically presenting an evaluation target having a randomized order by evaluating the randomness of the order of the evaluation targets. To provide a program.

In order to solve the above problems, the subjective evaluation device according to the present invention is a subjective evaluation device that presents random evaluation targets in the order of arrangement, and includes a sequence generator that generates a plurality of random sequences in the order of arrangement and a plurality of sequences. The evaluation target sorting unit that rearranges the evaluation targets based on the sequence and the evaluation target presentation unit that presents the evaluation targets sorted by the evaluation target sorting unit are provided, and the sequence generation unit is provided, respectively. In the sequence generator that generates a plurality of sequences with different sequences, the sequence recording unit that records the sequence, the reference sequence that is a sequence selected from the sequences recorded in the sequence recording unit, and the sequence recording unit. A common part number sequence detection unit that detects a common part number sequence that is a number sequence that has a partially common order between the detected number sequence that is one or more selected numbers from the recorded number sequences, and the common part. It is provided with a sequence selection output unit that selects and outputs one or more sequences from the detected sequences for which the common sequence is not detected by the sequence detection unit, and adds the sequence to the reference sequence. It is a feature.

Further, in the subjective evaluation device according to the present invention, in the sequence selection output unit, among the detected sequences, the length of the intersection sequence exceeds the threshold value, or the number of appearances of the intersection sequence exceeds the threshold value. Is excluded, and one or more sequences are selected from the remaining sequences to be detected.

Further, in the subjective evaluation device according to the present invention, when the common sequence detection unit detects a common sequence in all of the detected sequences, the sequence generator newly generates a sequence and the sequence is generated. Each time the sequence of the recording unit is updated or the sequence selection output unit outputs the sequence, a new sequence is generated and the sequence of the sequence recording unit is updated.

Further, in the subjective evaluation device according to the present invention, when the common partial sequence detection unit detects the common partial sequence between the reference sequence and the detected sequence, the reference sequence and the detected sequence are detected. The sequence selection output unit obtains the degree of overlap in the order of the above, and is characterized in that the sequence having the minimum degree of overlap is selected and output.

を用いて前記重複度を算出することを特徴とする。 Further, in the subjective evaluation device according to the present invention, the common part number sequence detection unit is shown in the following equation with the length l of the common part number sequence and the number of appearances ml of the common part number sequence of length _l as parameters. Evaluation function E

It is characterized in that the overlap degree is calculated by using.

Further, in the subjective evaluation device according to the present invention, the sequence generation unit receives the detection result and the detected number sequence output by the common part number sequence detection unit, and the detected number sequence in which the common part number sequence is not detected, or the overlap degree. When there are a plurality of detected sequences that are less than or equal to the threshold value, between the edited distance measurement sequence selection unit that selects the detected sequence as the edited distance measurement sequence, and the reference sequence and the edited distance measurement sequence. The sequence selection output unit further includes an edit distance measurement unit for measuring the edit distance of the above, and is characterized in that the sequence selection output unit selects and outputs the sequence to be edited distance measurement having the maximum edit distance.

Further, in the subjective evaluation device according to the present invention, the editing distance measuring unit is characterized in that the operation of the editing distance is targeted only for replacement and movement for each number in units of one number or block.

Further, in the subjective evaluation device according to the present invention, it is desirable that the sequence generation unit inputs a sequence to be excluded or a sequence of intersections in advance and prohibits the generation of the sequence.

Further, in the subjective evaluation device according to the present invention, the sequence generator classifies the number of generation targets into a plurality of groups according to categories, and generates a sequence using at least one of stratified randomization and adaptive randomization. It is desirable to let it.

Further, in order to solve the above-mentioned problems, the program according to the present invention is characterized in that the computer functions as the above-mentioned subjective evaluation device.

According to the present invention, it is possible to automatically present an evaluation target having a randomized order.

It is a block diagram which shows the structural example of the subjective evaluation apparatus which concerns on 1st Embodiment of this invention. It is a block diagram which shows the structural example of the sequence generation part of the subjective evaluation apparatus which concerns on 1st Embodiment of this invention. It is a figure which shows the detection example in the intersection sequence detection part of the subjective evaluation apparatus which concerns on 1st Embodiment of this invention. It is a figure which shows the pseudo MATLAB code for the intersection sequence detection part in the intersection sequence detection part of the subjective evaluation apparatus which concerns on 1st Embodiment of this invention. It is a figure which shows the detection result by the pseudo-MATLAB code shown in FIG. It is a figure which shows the detection example of the degree of overlap in the intersection sequence detection part of the subjective evaluation apparatus which concerns on 1st Embodiment of this invention. It is a block diagram which shows the structural example of the subjective evaluation apparatus which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the structural example of the sequence generation part of the subjective evaluation apparatus which concerns on 2nd Embodiment of this invention. It is a figure which shows the measurement example of the block edit distance measured by the edit distance measurement unit of the subjective evaluation apparatus which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the operation example of the sequence generation in the subjective evaluation apparatus which concerns on 2nd Embodiment of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First Embodiment)
FIG. 1 is a diagram showing a configuration example of a subjective evaluation device according to the first embodiment of the present invention. The subjective evaluation device 1 is a device that presents a random evaluation target in the order of arrangement to the evaluator, and includes a sequence generation unit 10, an evaluation target recording unit 20, an evaluation target sorting unit 30, and an evaluation target presentation unit 40. , The evaluation result recording unit 50 is provided.

The evaluation target recording unit 20 records a plurality of evaluation targets (for example, document data, image data, audio data, etc.) to be subject to subjective evaluation.

The sequence generation unit 10 inputs the length l of the sequence and the required number x (x ≧ 1) of the sequence, generates x sequences of random length l in the order of arrangement, and the evaluation target sorting unit 30. Output to. Details will be described later.

The evaluation target sorting unit 30 acquires a plurality of evaluation targets from the evaluation target recording unit 20, sorts and sorts the acquired plurality of evaluation targets based on the number columns generated by the number sequence generation unit 10, and sorts the evaluation targets. Is output to the evaluation target presentation unit 40. For example, when there are five evaluation targets A, B, C, D, and E acquired from the evaluation target recording unit 20, and the sequence generated by the sequence generation unit 10 is [3 1 5 2 4]. , A, B, C, D, E are rearranged in the third, first, fifth, second, and fourth positions, respectively. Alternatively, A, B, C, D, and E may be assigned 1 to 5 respectively, and the sequences generated by the sequence generation unit 10 may be rearranged in this order. In that case, if the sequence generated by the sequence generation unit 10 is [3 1 5 2 4], the evaluation targets are sorted in the order of C, A, E, B, D.

The evaluation target presentation unit 40 presents the evaluation target sorted by the evaluation target sorting unit 30 to the evaluator. For example, when the evaluation target is document data or image data, the evaluation target is displayed on the display, and when the evaluation target is audio data, the evaluation target is output from the speaker. The evaluation target presentation unit 40 presents the evaluation targets to the evaluator one by one, and the evaluator evaluates the evaluation targets one by one.

The evaluation result recording unit 50 records the evaluation result of the evaluation target by the evaluator.

Next, the sequence generation unit 10 will be described in detail.

FIG. 2 is a diagram showing a configuration example of the sequence generation unit 10. The sequence generation unit 10 includes a sequence generation unit 11, a sequence recording unit 12, an intersection sequence number sequence detection unit 13, and a sequence selection output unit 14.

The sequence generation unit 11 inputs the length of the sequence (the number of numbers, here n) and the required number x of the sequence, and generates a plurality of sequences having different order sequences. A known pseudo-random number generation algorithm can be used to generate the sequence. For example, the sequence generation unit 11 classifies the number of generation targets into a plurality of groups according to categories, and generates a sequence using at least one of stratified randomization and adaptive randomization (minimization). By using these randomization methods, it is possible to balance the selection frequency among a plurality of categories and control the selection so that the selection is not concentrated in the same category, which is suitable for randomization.

Further, the sequence generation unit 11 can prohibit the generation of the sequence by inputting the sequence to be excluded or the intersection sequence in advance. By doing so, when the recording order in the evaluation target recording unit 20 is known, the order of the evaluation targets that tends to lead to biased results in subjective evaluation (groups with a small difference or groups belonging to a specific category). It is suitable for randomization because it can avoid continuous).

The sequence recording unit 12 records the sequence of x length n generated by the sequence generator 11.

The intersection sequence detection unit 13 randomly selects one sequence from the sequences recorded in the sequence recording unit 12 and deletes it from the sequence recording unit 12 to obtain a reference sequence, and generates the reference sequence. Output from unit 10. Then, a sequence in which the order is partially common between the reference sequence and one or more detected sequences arbitrarily selected from the sequences recorded in the sequence recording unit 12 (hereinafter, "intersection sequence"). ”) Is detected and output to the sequence selection output unit 14.

The sequence selection output unit 14 has one or more of the detected sequences from which the intersection sequence was not detected when there is a detected sequence in which the intersection sequence detection unit 13 did not detect the intersection sequence. Select a sequence and output it.

The sequence selection output unit 14 excludes the detected sequences whose length of the intersection sequence exceeds the threshold and / or the number of appearances of the intersection sequence exceeds the threshold, and excludes the remaining sequences of the detected sequences. One or more sequences may be selected and output. For example, when there is only one detected sequence for which the common sequence detection unit 13 has not detected the common sequence, the sequence selection output unit 14 outputs the detected sequence for which the common sequence has not been detected. , Of the detected sequences in which the common partial sequence is detected, the detected sequence in which the length and the number of appearances of the common partial sequence are equal to or less than the threshold may be output.

Further, when the intersection sequence is detected between the reference sequence and the detected sequence, the intersection sequence detection unit 13 has a degree of overlap in the order (numbers) between the reference sequence and the detected sequence. Adjacent frequency) may be calculated.

When the intersection sequence detection unit 13 calculates the degree of duplication, the sequence selection output unit 14 selects and outputs one or more sequences from the detected sequences in ascending order of the degree of overlap.

When the intersection sequence detection unit 13 detects the intersection sequence in all the detected sequences, the sequence generation unit 11 newly generates a sequence, and the intersection sequence detection unit 13 does not detect the intersection sequence. Until then, the sequence of the sequence recording unit 12 may be updated. Similarly, when the degree of duplication exceeds the threshold value, the sequence generation unit 11 may generate a new sequence and update the sequence of the sequence recording unit 12.

Further, the sequence generation unit 11 may generate a new sequence every time the sequence selection output unit 14 outputs the sequence, and may update the sequence of the sequence recording unit.

The intersection sequence detection unit 13 includes a reference sequence storage unit 131, a detected sequence storage unit 132, and a detection unit 133.

As a process limited to the first time, the reference sequence storage unit 131 acquires an arbitrary sequence _ai recorded in the sequence recording unit 12, deletes the sequence _ai from the sequence recording unit 12, and stores it as a reference sequence. At the same time, it is output from the sequence generation unit 10.

The detected sequence storage unit 132 acquires an arbitrary number of sequences from the sequences other than the sequence _ai recorded in the sequence recording unit 12, and stores the sequence as a detected sequence.

The detection unit 133 detects the intersection sequence between the reference sequence and the detected sequence in a round-robin manner, and outputs the detection result to the sequence selection output unit 14.

When the detection unit 133 detects the intersection sequence, it calculates the degree of overlap (adjacent frequency of numbers) in the order of the reference sequence and the sequence of detected sequences, and associates the degree of overlap with each sequence of detected sequences. It may be output to the sequence selection output unit 14. Specifically, the length l of the intersection sequence and the number (number of appearances) mL of the intersection sequence of length _l are stored. Then, the degree of duplication is calculated by defining the evaluation function E _l for the intersection sequence of length l with l and _mL as parameters as shown in the equation (1).

FIG. 3 shows a detection example of the intersection sequence detection unit 13 for a set of sequences. Here, the number of times detected for each length of the intersection sequence is recorded for the reference sequence [1 2 3 4 5 6 7 8 9 10] and the detected sequence [3 4 5 6 9 1 2 10 7 8]. , Define the evaluation function E _l . In this example, there is an intersection sequence [3 4 5 6] of length l = 4 between the reference sequence and the detected sequence (number of appearances m ₄ = 1), and the intersection of length l = 2. There are sequences [1 2] and [78] (number of appearances m ₂ = 2).

The degree of overlap with respect to the intersection sequence can be quantified by using, for example, the evaluation function E ^ which is the sum of the equation (1) with respect to l shown in the equation (2). In the example shown in FIG. 3, E ^ = E ₂ + E ₄ = 2 + 3 = 5.

The evaluation function E ^ is not limited to the equation (2), and can be evaluated by the size of the length _l or the number of appearances ml, and the length _l or the number of appearances ml is an exponential. It can also be treated as an exponential of a function. Further, the maximum value of the length l is n-1, because it is premised that the sequence generation unit 11 does not generate a sequence that completely matches each other. Therefore, if there is a possibility of generating sequences that completely match each other depending on the algorithm of the sequence generation unit 11, the maximum value of the length l may be set to n.

FIG. 4 shows an example of a pseudo MATLAB code (registered trademark) for detecting an intersection sequence. It is a mechanism to store the location and length of the intersection sequence in scoreTab.

FIG. 5 shows the detection result by the pseudo MATLAB code shown in FIG. scoreTab is a mechanism that increments the length of the intersection sequence in the diagonally lower right direction. If there is only one, it is not an intersection sequence because only one number is duplicated. From FIG. 5, it was detected that two intersection sequences of length l = 2 (number of appearances m _l = 2) and one intersection sequence of length l = 4 (number of appearances _ml = 1). I understand.

The sequence selection output unit 14 selects the sequence having the smallest degree of overlap and outputs it to the evaluation target sorting unit 30. The sequence selection output unit 14 sends the output information of the detected sequence to the sequence recording unit 12, and the sequence is deleted from the sequence recording unit 12. Further, the sequence selection output unit 14 stores the output detected sequence in the reference sequence storage unit 131 and adds it to the reference sequence. As a result, the sequence to be output next by the sequence selection output unit 14 becomes a sequence having a small degree of duplication with respect to all the sequences output in the past.

Further, the sequence selection output unit 14 sends the output sequence information and the detected sequence information whose degree of duplication exceeds the threshold to the detected sequence storage unit 132, and the sequence is deleted from the detected sequence storage unit 132. , The output sequence is additionally stored in the reference sequence storage unit 131. It should be noted that the storage area for the number sequence deleted from the detected number sequence storage unit 132 may be supplemented with an additional number sequence from the number sequence recording unit 12, or the entire number sequence of the detected number sequence storage unit 132 may be updated each time. May be good.

In outputting the second and subsequent sequences, the intersection sequence detection unit 13 is between the plurality of sequences stored in the reference sequence storage unit 131 and the plurality of sequences stored in the detected sequence storage unit 132. A round-robin evaluation is made for the intersection sequence. As in the previous process, the sequence selection output unit 14 determines the sequence to be output from the brute force detection result.

FIG. 6 shows a combination of all the sequences input to the intersection sequence detection unit 13 and an example of detecting the degree of duplication. It is assumed that the sequence r _i is stored in the reference sequence storage unit 131 and the sequence p _i is stored in the detected sequence storage unit 132. Since the detection is brute force as described above, the multiplicity E ^ _pi of each detected sequence is expressed by the equation (3).

The sequence selection output unit 14 determines the sequence to be output using the overlap degree E ^ _pi detected by the intersection sequence detection unit 13. In the example of FIG. 6, the sequence p ₂ having the lowest multiplicity E ^ _pi is output.

The present invention can be applied not only to a sequence but also to a character string. Further, in the case of a sequence in which the same number is used multiple times, the method of the present invention can be applied by assigning different numbers for convenience such as [1 1 2 2] to [1 2 3 4]. can.

Although the subjective evaluation device 1 has been described above, it is also possible to use a computer to function as the subjective evaluation device 1. Such a computer is realized by storing a program describing the processing contents that realize each function of the subjective evaluation device 1 in the storage unit of the computer, and reading and executing this program by the CPU of the computer. be able to. The program may also be recorded on a computer-readable medium. It can be installed on a computer using a computer-readable medium. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, but may be, for example, a recording medium such as a CD-ROM or a DVD-ROM.

As described above, in the first embodiment, the intersection sequence is detected between the reference sequence and the detected sequence, a sequence having a low degree of duplication is selected, and the evaluation target is based on the sequence having a low degree of duplication. And sort. With such a configuration, the subjective evaluation device 1 can automatically generate an evaluation target in which the arrangement order is randomized, that is, the degree of overlap is low and the arrangement order is small.

(Second embodiment)
Next, the subjective evaluation device according to the second embodiment of the present invention will be described.

FIG. 7 is a diagram showing a configuration example of the subjective evaluation device according to the second embodiment of the present invention. The subjective evaluation device 2 according to the second embodiment is different from the subjective evaluation device 1 according to the first embodiment in that it includes a sequence generation unit 10'instead of the sequence generation unit 10. Since the other configurations are the same, the sequence generation unit 10'will be described below.

FIG. 8 is a diagram showing a configuration example of the sequence generation unit 10'. The sequence generation unit 10'has a sequence generation unit 11, a sequence recording unit 12, a common portion sequence detection unit 13, a sequence selection output unit 14, an edited distance measurement sequence selection unit 15, and an edit distance measurement unit 16. To prepare for. The sequence generation unit 10'is different from the sequence generation unit 10 shown in the first embodiment in that it further includes an edited distance measurement sequence selection unit 15 and an edit distance measurement unit 16.

The sequence generation unit 10'selects the sequence in consideration of the editing distance as well as the intersection sequence. Here, the edit distance is the number of operations required to match one sequence with the other sequence by inserting / deleting / replacing numbers in the sequence, and the greater the number, the farther the distance. Become.

The intersection sequence detection unit 13 outputs the reference sequence in the reference sequence storage unit 131 to the editing distance measurement unit 16. Further, the intersection sequence detection unit 13 outputs the detected sequence and the degree of overlap, and outputs the output to the edited distance measurement sequence selection unit 15.

Based on the detection result of the common part number sequence detection unit 13, the edited distance measurement number sequence selection unit 15 has a plurality of detected number sequences in which the common part number sequence is not detected, or the detected number sequence having the same degree of overlap at the minimum or less than the threshold value. If there are a plurality of such sequences, the detected sequence is selected and output to the editing distance measuring unit 16 as the edited sequence. On the other hand, when there is only one detected sequence in which the common partial sequence is not detected, or the detected sequence having the minimum or less than the threshold, the edited distance measurement sequence selection unit 15 passes the edit distance measurement itself. Then, the detected sequence is output to the sequence selection output unit 14.

The editing distance measuring unit 16 measures the editing distance between the reference number sequence and the edited distance measurement number sequence, and outputs the measurement result and the edited distance measurement number sequence to the number column selection output unit 14.

The sequence selection output unit 14 selects and outputs the edited distance measurement sequence having the maximum edit distance.

The existing edit distances include Levenshtein distance (insert / delete / replace 1 number), Damerau-Levenshtein distance (insert / delete / replace / move 1 number), and edit distance with movement (insert / delete / replace 1 number).・ Movement of multiple numbers), etc., and it is also possible to apply these. However, the insertion and deletion are not suitable as an editing operation for the generation of a sequence in which the numbers do not overlap and the length is constant. Therefore, it is preferable that the editing distance measuring unit 16 performs the editing distance operation only for the replacement and movement of one number or each number in the block unit. Such an editing distance for only replacement and movement of one number or each number in a block unit is hereinafter referred to as "block editing distance".

FIG. 9 shows a measurement example of the block editing distance measured by the editing distance measuring unit 16. When the reference sequence is [1 2 3 4 5 6 7 8 9 10] and the edited distance measurement sequence is [3 4 5 6 9 1 2 10 7 8], from the reference sequence to the edited distance measurement sequence. The measurement process of the block edit distance is shown. In this example, in the first edit, the number [3 4 5 6] in block units is moved with respect to the reference sequence. In the second edit, the number [78] in block units is moved with respect to the sequence of the first edit. In the third edit, the number 1 number [9] in the block unit is moved to the sequence of the second edit. Since it matches the edited distance measurement sequence in the third edit, the block edit distance in this case is "3".

FIG. 10 is a flowchart showing an operation example of sequence generation in the sequence generation unit 10'. First, the sequence generator 11 acquires the length and number of sequences to be generated (step S101), and generates a sequence according to the conditions (step S102).

If the reference sequence is not stored in the reference sequence storage unit 131 (step S103-No), the reference sequence is selected from the generated sequences and stored in the reference sequence storage unit 131 (step S104). Then, the detection unit 133 detects the intersection sequence (step S105). On the other hand, when the reference sequence is stored in the reference sequence storage unit 131 (step S103-Yes), step S104 is skipped and the intersection sequence is detected by the detection unit 133 (step S105).

As a result of detecting the intersection sequence in step S105, if there is one sequence having the minimum degree of overlap (step S106-Yes), the sequence is output as it is (step S109).

On the other hand, when the common partial sequence is not detected, or when there are a plurality of detected sequences whose degree of duplication is equal to or less than the threshold value, or when there is not one sequence having the minimum degree of duplication (step S106-No), the edited distance. The measurement sequence selection unit 15 selects them as the edited distance measurement sequence (step S107), and the edit distance measurement unit 16 measures the edit distance from the reference sequence (step S108). As a result, the sequence selection output unit 14 outputs the sequence having the largest edit distance, stores the sequence in the reference sequence storage unit 131, and adds the sequence to the reference sequence (step S109).

If a predetermined number of sequences has been generated by this point (step S110-Yes), the generation process is completed, and if the number is less than the predetermined number (step S110-No), the intersection sequence Returning to the detection (step S105), the processes from step S105 to step S109 are repeated until the generation of a predetermined number of sequences is completed.

Although the subjective evaluation device 2 has been described above, it is also possible to use a computer to function as the subjective evaluation device 2. Such a computer is realized by storing a program describing the processing contents that realize each function of the subjective evaluation device 2 in the storage unit of the computer, and reading and executing this program by the CPU of the computer. be able to. The program may also be recorded on a computer-readable medium. It can be installed on a computer using a computer-readable medium. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, but may be, for example, a recording medium such as a CD-ROM or a DVD-ROM.

As described above, in the second embodiment, when the common partial sequence is not detected between the reference sequence and the detected sequence, or when there are a plurality of detected sequences whose degree of duplication is equal to or less than the threshold value, the reference is used. The edit distance is measured between the sequence and the edit distance measurement sequence, and the edit distance measurement sequence with the maximum edit distance is selected and output. With this configuration, even if the number sequence with the minimum degree of duplication is not one, the sequence selection output unit 14 can select the sequence, and the subjective evaluation device 2 automatically selects the evaluation target whose order is randomized. Can be generated as a target.

Although the above embodiments have been described as typical examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications and modifications can be made without departing from the scope of claims. For example, it is possible to combine a plurality of the constituent blocks described in the configuration diagram of the embodiment into one, or to divide one constituent block into one.

1,2 Subjective evaluation device 10,10'Sequential sequence generation unit 11 Sequence generation unit 12 Sequence recording unit 13 Common part Sequence detection unit 14 Sequence selection output unit 15 Edited distance measurement Sequence selection unit 16 Editing distance measurement unit 20 Evaluation target recording unit 30 Evaluation target sorting unit 40 Evaluation target presentation unit 50 Evaluation result recording unit 131 Reference sequence storage unit 132 Detected sequence storage unit 133 Detection unit

Claims (8)

It is a subjective evaluation device that presents random evaluation targets in the order of arrangement.
A sequence generator that generates multiple random sequences in the order of arrangement,
An evaluation target sorting unit that sorts a plurality of evaluation targets based on the sequence,
It is provided with an evaluation target presentation unit that presents the evaluation target sorted by the evaluation target sorting unit.
The sequence generator is
A sequence generator that generates multiple sequences with different order sequences,
A sequence recording unit that records the sequence and a sequence recording unit
A portion between the reference sequence, which is a sequence selected from the sequences recorded in the sequence recording unit, and the detected sequence, which is one or more sequences selected from the sequences recorded in the sequence recording unit. A common part number sequence detector that detects a common part number sequence that is a number sequence with a common order.
A sequence selection output unit that selects and outputs one or more sequences from the detected sequences for which the intersection sequence detection unit did not detect the intersection sequence, and adds the sequence to the reference sequence.
A subjective evaluation device, characterized in that it is provided with. The subjective evaluation device according to claim 1.
The sequence selection output unit excludes the detected sequences whose length exceeds the threshold, or the number of appearances of the common sequence exceeds the threshold, and is among the remaining detected sequences. A subjective evaluation device characterized by selecting one or more sequences from. The subjective evaluation device according to claim 1 or 2.
When the common part number sequence detection unit detects a common part number sequence in all of the detected number sequences, the sequence generation unit generates a new sequence and updates the sequence of the sequence recording unit, or A subjective evaluation device, characterized in that each time the sequence selection output unit outputs a sequence, a new sequence is generated and the sequence of the sequence recording unit is updated. The subjective evaluation device according to claims 1 to 3.
When the intersection sequence detection unit detects the intersection sequence between the reference sequence and the detected sequence, the intersection sequence detection unit obtains the degree of overlap in the order of the reference sequence and the detected sequence.
The sequence selection output unit is a subjective evaluation device characterized in that it selects and outputs a sequence having the minimum degree of duplication. The subjective evaluation device according to claim 4.
The evaluation function E shown in the following equation is used in the common sub-sequence detection unit with the length l of the common sub-sequence and the number of appearances m _l of the common sub-sequence having the length l as parameters.

A subjective evaluation device, characterized in that the degree of duplication is calculated using the above. The subjective evaluation device according to claim 4 or 5.
The sequence generator is
When the detection result output by the common partial sequence detection unit and the detected sequence are received and the common partial sequence is not detected, or when there are a plurality of detected sequences whose degree of duplication is equal to or less than the threshold value, the subject is detected. The edited distance measurement sequence selection unit that selects the detected sequence as the edited distance measurement sequence, and
Further, an editing distance measuring unit for measuring an editing distance between the reference sequence and the edited distance measurement sequence is provided.
The sequence selection output unit is a subjective evaluation device characterized in that it selects and outputs a sequence of measured distances to be edited having the maximum edit distance. The subjective evaluation device according to claim 6.
The editing distance measuring unit is a subjective evaluation device characterized in that the operation of the editing distance is performed only for replacement and movement of one number or each number in a block unit. A program for operating a computer as the subjective evaluation device according to any one of claims 1 to 7.

Images