JP7687087B2 - Information processing device, information processing method, and computer program - Google Patents

Description

The present invention relates to an information processing device that arranges and outputs multiple microcontents.

Conventionally, newspaper and magazine articles distributed in paper form are distributed to computers and displayed on computer screens. Articles and the like distributed to computers are called digital content. Furthermore, each of the multiple pieces of text data and images that make up digital content is called microcontent. Because computer screens come in a variety of sizes, in order to display digital content on a computer screen in an easy-to-view manner, layout information is required that specifies how the microcontent that makes up the digital content should be arranged. The layout information determines the size and display position on the screen that matches the content of the microcontent.

One type of digital content is learning materials. Learning materials can be a combination of multiple microcontents such as text and charts, and their content varies widely. For this reason, when providing learning materials as digital content, layout information needs to be created for each piece of digital content, which is cumbersome. In response to this situation, a digital content display method has been proposed that automatically selects or generates a display layout, making it possible to display digital content in a layout that corresponds to the nature of the page information and the user's requirements, without the content creator having to design the screen configuration for each individual page (Patent Document 1).

JP 2003-67100 A

However, the technology disclosed in Patent Document 1 treats questions including diagrams and tables as a single element, and does not provide sufficient layout flexibility. The present invention has been made in light of this situation. Its purpose is to provide an information processing device or the like that can rearrange multiple questions and answer areas included in learning materials.

The information processing device according to one aspect of the present application is characterized by having an acquisition unit that acquires multiple pieces of question data that associate questions with answer columns, a generation unit that generates a teaching material image in which each piece of question data is arranged according to the page size and the layout size of the question data, and an output unit that outputs the generated teaching material image.

According to one aspect of the present application, it is possible to rearrange multiple question and answer areas contained in the learning material.

FIG. 1 is an explanatory diagram illustrating an example of the configuration of a teaching material generation system. FIG. 2 is a block diagram showing an example of a hardware configuration of a generation server. FIG. 2 is a block diagram showing an example of a hardware configuration of a user terminal. FIG. 13 is an explanatory diagram showing an example of a main question DB. FIG. 13 is an explanatory diagram showing an example of a short question DB. FIG. 11 is an explanatory diagram showing an example of a placement template. FIG. 11 is an explanatory diagram illustrating an example of a template DB. FIG. 11 is an explanatory diagram illustrating an example of a user setting DB. 13 is a flowchart illustrating an example of a generating process procedure. 13 is a flowchart illustrating an example of a procedure for a template selection process. FIG. 13 is an explanatory diagram showing an example of a problem to be placed. FIG. 13 is an explanatory diagram illustrating the process of determining whether a question for a main question fits within a certain area. FIG. 11 is an explanatory diagram illustrating a process for determining whether a chart can be placed. FIG. 11 is an explanatory diagram showing an example of a generated image. FIG. 11 is an explanatory diagram showing an example of a placement template. FIG. 11 is an explanatory diagram showing an example of a placement template. FIG. 2 is an explanatory diagram showing an example of an input image. FIG. 11 is an explanatory diagram showing a result of area recognition. FIG. 11 is an explanatory diagram showing an example of an OCR result. FIG. 11 is an explanatory diagram showing the correspondence relationship between areas. 13 is a flowchart showing an example of a procedure for question data creation processing. FIG. 1 is an explanatory diagram illustrating an example of a learning model. 4 is a block diagram showing an example of a functional configuration of a generation server. FIG.

(Embodiment 1)
An embodiment will be described below with reference to the drawings. Fig. 1 is an explanatory diagram showing a configuration example of a teaching material generation system. The teaching material generation system 100 includes a generation server 1 and a user terminal 2. The generation server (information processing device) 1 and the user terminal 2 are connected to each other via a network N so as to be able to communicate with each other. In Fig. 1, only two user terminals 2 are shown, but there may be only one, or three or more user terminals.

The generation server 1 is configured with a server computer, a PC (Personal Computer), etc. The generation server 1 may also be configured with a multi-computer consisting of multiple computers, a virtual machine virtually constructed by software, or a quantum computer. Furthermore, the functions of the generation server 1 may be realized by a cloud service. The user terminal 2 is configured with a notebook computer, a tablet computer, a smartphone, etc.

Figure 2 is a block diagram showing an example of the hardware configuration of the generation server. The generation server 1 includes a control unit 11, a main memory unit 12, an auxiliary memory unit 13, a communication unit 14, and a reading unit 15. The control unit 11, the main memory unit 12, the auxiliary memory unit 13, the communication unit 14, and the reading unit 15 are connected by a bus B.

The control unit 11 has one or more arithmetic processing devices such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), etc. The control unit 11 reads out and executes a control program 1P (program, program product) stored in the auxiliary storage unit 13, thereby performing various information processing, control processing, etc. related to the generation server 1.

The main memory unit 12 is a static random access memory (SRAM), a dynamic random access memory (DRAM), a flash memory, etc. The main memory unit 12 mainly temporarily stores data required for the control unit 11 to execute arithmetic processing.

The auxiliary memory unit 13 is a hard disk or SSD (Solid State Drive) or the like, and stores the control program 1P and various DBs (databases) necessary for the control unit 11 to execute processing. The auxiliary memory unit 13 stores a main question DB 131, a sub-question DB 132, a template DB 133, and a user setting DB 134. The auxiliary memory unit 13 may be an external storage device that is separate from and externally connected to the generation server 1. The various DBs and the like stored in the auxiliary memory unit 13 may be stored in a database server or cloud storage different from the generation server 1.

The communication unit 14 communicates with the user terminal 2 via the network N. The control unit 11 may also use the communication unit 14 to download the control program 1P from another computer via the network N, etc., and store it in the auxiliary storage unit 13.

The reading unit 15 reads portable storage medium 1a, including CD (Compact Disc)-ROM and DVD (Digital Versatile Disc)-ROM. The control unit 11 may read the control program 1P from the portable storage medium 1a via the reading unit 15 and store it in the auxiliary storage unit 13. The control unit 11 may also read the control program 1P from the semiconductor memory 1b.

Figure 3 is a block diagram showing an example of the hardware configuration of a user terminal. The user terminal 2 includes a control unit 21, a main memory unit 22, an auxiliary memory unit 23, a communication unit 24, an input unit 25, and a display unit 26. Each component is connected by a bus B.

The control unit 21 has one or more arithmetic processing devices such as a CPU, MPU, GPU, etc. The control unit 21 provides various functions by reading and executing the control program 2P stored in the auxiliary storage unit 23.

The main memory unit 22 is an SRAM, DRAM, flash memory, etc. The main memory unit 22 mainly temporarily stores data required for the control unit 21 to execute arithmetic processing.

The auxiliary storage unit 23 is a hard disk or SSD, etc., and stores various data necessary for the control unit 21 to execute processing. The auxiliary storage unit 23 may be an external storage device connected to the user terminal 2. The various DBs, etc. stored in the auxiliary storage unit 23 may be stored in a database server or cloud storage.

The communication unit 24 communicates with the generation server 1 via the network N. The control unit 21 may also use the communication unit 24 to download the control program 2P from another computer via the network N, etc., and store it in the auxiliary storage unit 23.

The input unit 25 is a keyboard and a mouse. The display unit 26 includes a liquid crystal display panel, etc. The display unit 26 displays teaching materials, etc. output by the generation server 1. The display unit 26 and the input unit 25 may be integrated to form a touch panel display. The user terminal 2 may display on an external display device.

Next, the database used by the teaching material generation system 100 will be explained. In the following explanation, it is assumed that the teaching material is composed of main questions (groups), and each main question contains sub-questions. It is also assumed that the teaching material has an area in which answers are written, like tests and drills used in elementary and junior high schools. In addition, mathematics teaching materials will be used to illustrate the data structure and processing content, but they can also be applied to other subjects.

Figure 4 is an explanatory diagram showing an example of a large question DB. Large question DB 131 stores information about large questions. Large question DB 131 includes a large question ID column, a data type column, a number of characters/number of images column, a question column, a number of sub-questions column, a number of figures/tables column, a sequence number column, and a sub-question ID column. The large question ID column stores a large question ID that uniquely identifies a large question. The large question ID is generated when a record is added to large question DB 131. The data type column stores the data type of the large question stored. The data type is, for example, text or image. The type being text indicates that the question is stored as text. The type being image indicates that the question is stored as an image. The number of characters/number of images column stores the amount of the question. If the data type is text, the number of characters/number of images column stores the number of characters of the question (number of characters of text). If the data type is image, the number of characters/number of images column stores the size of the question image (image size). The question string stores the question. If the data type is text, the question string stores the text data of the question. If the data type is image, the question string stores the question image. The question string may store the image file name of the question image. The number of sub-questions string stores the number of sub-questions (number of questions and answers) included in the large question. The number of figures and tables string stores whether or not the large question includes a figure. If the large question includes a figure, the number of figures and tables string stores the number of figures and tables (number of images). If the large question does not include a figure, the number of figures and tables string stores 0. The figure and table string stores the entity data or identification information (ID or file name) of the figure. The sequence number string and the sub-question ID string store the sequence number (sequence information) and sub-question ID for the sub-questions included in the large question, respectively. Note that in this specification, the question is not limited to text, but also includes mathematical formulas for calculation problems.

Figure 5 is an explanatory diagram showing an example of a sub-question DB. The sub-question DB 132 stores information about sub-questions. The sub-question DB 132 includes a sub-question ID column, a data type column, a type column, a character count/image count column, and a content column. The sub-question ID column stores a sub-question ID that uniquely identifies a sub-question. The sub-question ID is generated when a record is added to the sub-question DB 132. The data type column stores the data type of the stored sub-question. The data type is the same as that of the main question. The type column stores a question or an answer. The type being question indicates that the data following the character count/image count column is related to the question. The type being answer indicates that the data following the character count/image count column is related to the answer section. The character count/image count column is the same as the character count/image count column in the main question DB 131, so an explanation is omitted. As for the content column, when the data type is text and the type column is question, the content column stores the text data that constitutes the question portion of the sub-question. If the data type is text and the type column is answer, the content column stores the text data that makes up the answer section of the sub-question. The text data mainly consists of blank characters with a decorative frame. If the data type is image and the type column is question, the content column stores the image or image file name that makes up the question section of the sub-question. If the data type is image and the type column is answer, the content column stores the image or image file that makes up the answer section of the sub-question. The image mainly consists of a frame shape.

Note that main question DB131 and sub-question DB132 are examples of storage units that store attributes. Main question DB131 stores correspondence information between main question IDs and diagrams. The correspondence information is an example of association information between questions and diagrams. Sub-question DB132 stores correspondence information between questions and answer columns for each sub-question. The correspondence information is an example of association information between questions and answer columns. The content column of sub-question DB132 includes information such as (1), (2), etc. Numbers without parentheses are an example of question/answer number information.

Figure 6 is an explanatory diagram showing an example of a layout template. A layout template specifies how the question text of a main question, the question parts of sub-questions, the answer columns of the sub-questions, and diagrams and charts, etc., should be laid out on a computer screen. Figure 6 illustrates the layout template so that it can be visually understood. Figure 6 shows six patterns, but is not limited to these. Hereinafter, a "layout template" will also be referred to simply as a "template."

Figure 7 is an explanatory diagram showing an example of a template DB. Template DB 133 is a relational database of the placement template described with reference to Figure 6. Template DB 133 includes a template ID column, a configuration column, an element column, a position column, and a size column. The template ID column stores a template ID that uniquely identifies the placement template. The template ID is generated when a record is added to template DB 133. The configuration column stores the number of elements to be placed in the placement template. The elements are main questions, sub-questions, answer columns, and diagrams. The placement template with template ID = 1 shown in Figure 7 indicates that two main questions, seven sub-questions, seven answer columns, and one diagram are placed. The element column stores which element the data after the position column is for. The position column stores coordinate values indicating the placement position of each element. The size column indicates the size of the area that each element can occupy. Here, it is assumed that each element is placed in a rectangular area. The position column stores the coordinate value of the upper left corner of the rectangular area. The size column stores the width and height (placement size) of a rectangular area. Alternatively, the coordinate values of the four corners of a rectangular area may be stored to express the position and size of the area. Note that the template DB 133 shown in FIG. 7 does not include a column that stores the size of the entire template. The size of the entire template (page size) is determined separately. By providing the template DB 133 with a column that stores the overall size, it may be possible for the template DB 133 to store records of various placement templates with different overall sizes.

Figure 8 is an explanatory diagram showing an example of a user setting DB. The user setting DB 134 stores the settings for displaying questions for each user. The user setting DB 134 includes a user ID column, a main question column, a sub-question column, an answer column, a diagram column, a character spacing column, and a line spacing column. The user ID column stores a user ID. The user ID is an ID that can uniquely identify a user, and is determined when the user registers in the teaching material generation system. The main question column, the sub-question column, and the answer column store the settings for the main question, the sub-question, and the answer column, respectively. The main question column, the sub-question column, and the answer column each include a character size column and a font column. The character size column stores the size of the characters to be used. In Figure 8, the maximum and minimum character sizes are stored in points. The font column stores the character font to be used. The diagram column includes an aspect ratio difference column. The aspect ratio difference column stores the degree to which the aspect ratio change is allowed when fitting a diagram into the diagram area of the placement template. An aspect ratio difference column of ±1 means, for example, that a chart with an aspect ratio of 3:4 is allowed to be modified in the range of 2:3 to 4:5. The character spacing column stores the character spacing when displayed. Character spacing is expressed as a ratio to the character size, for example. 8% means that if the character size is 16 points, then character spacing of 1.28 points will be provided. The line spacing column stores the spacing between lines. 0.6 characters means that a spacing equivalent to 60% of the character size will be provided between lines.

Next, we will explain the information processing performed by the teaching material generation system 100. In the following explanation, it is assumed that text data has already been obtained for the main questions, sub-questions, and answers, excluding the chart data.

9 is a flowchart showing an example of the procedure of the generation process. The control unit 11 of the generation server 1 acquires the grouped problem data, the number of the problem to be placed, and the number of images n to be displayed (step S1). The combination of these data may be determined in advance and stored in the auxiliary storage unit 13. Alternatively, the control unit 11 may generate the combination of these data using a predetermined algorithm based on the progress of the learner who uses the teaching material, or the teacher who creates the teaching material may specify it. The control unit 11 selects a placement template based on the number of problems to be placed, the number of characters of the text, and the aspect ratio of the figures and tables (step S2). The control unit 11 refers to the main question DB 131 and the small question DB 132 and acquires the number of characters of the text of the main question and small question to be placed. Based on the contents of the figure column in the main question DB 131, image data of the figure to be placed is acquired, and the aspect ratio of the figure is acquired.

Figure 10 is a flowchart showing an example of the procedure of the template selection process. The control unit 11 refers to the template DB and selects a template whose number of questions (major questions, minor questions), answers, and figures and charts matches the number of questions, answers, and charts to be placed (step S21). The number of charts and charts is obtained by searching the major question DB 131 using the number of the question to be placed (major question ID) as a key. The control unit 11 selects a template whose number of characters and character size match the placement area size from among the templates selected in step S21 (step S22). The character size follows the user settings stored in the user setting DB 134. The control unit 11 selects a template whose aspect ratio (width:height) of the display area of the chart matches the chart to be placed from among the templates selected in step S22 (step S23). If no template is selected by executing step S23, an error occurs. In this case, the template selection process returns an error. If an error is returned, the control unit 11 stops the generation process and performs error processing. For example, an error message is displayed on the teacher's terminal that started the generation process. The error is also recorded in an error log, and an email is sent to the administrator to notify them of the error. The control unit 11 returns the template selected in step S23 to the caller of the process, with the template selected as the selection result.

The control unit 11 judges whether or not a plurality of templates are selected as a result of the template selection process (step S3). If the control unit 11 judges that a plurality of templates are selected (YES in step S3), the control unit 11 arranges questions, answers, and diagrams for each selected template and generates an image (step S4). The control unit 11 calculates the blank space ratio of the generated image and sorts the images according to the calculated blank space ratio (step S5). For example, the sorting order is the lowest blank space ratio. The control unit 11 outputs the top n images of the sorting result (step S6) and ends the process. If the control unit 11 judges that a plurality of templates are not selected (NO in step S3), the control unit 11 arranges questions, answers, and diagrams in the selected template and generates an image (step S7). The control unit 11 outputs the sorted image (step S6) and ends the process. For example, in step S7, the image is transmitted to the user terminal 2 of the user who is the learner.

Next, the generation process and template selection process will be explained using a concrete example. FIG. 11 is an explanatory diagram showing an example of questions to be placed. Here, two main questions are placed. Main question 11 includes a question statement 111 and four sub-questions 112. Each of the sub-questions 112 includes a question statement 113 and an answer column 114. Main question 12 includes a question statement 121, three sub-questions 122, and a diagram 125. Each of the sub-questions 122 includes a question statement 123 and an answer column 124. In the example shown in FIG. 11, two main questions, seven sub-questions, seven answer columns, and one diagram are placed.

When the control unit 11 executes step S21, the number of elements to be arranged is compared with the number of elements of each template shown in FIG. 6, and the templates of FIG. 6A, FIG. 6B, FIG. 6C, FIG. 6D, and FIG. 6E are selected. When the control unit 11 executes step S22, the settings stored in the user setting DB 134 are referenced, and it is determined whether the question text, etc., fits in each area with the specified character size. FIG. 12 is an explanatory diagram illustrating the process of determining whether the question text of the main question fits. The question text of the main question is applied to the template shown in FIG. 6E. FIG. 12A shows the case of question text 111 of main question 11, and FIG. 12B shows the case of question text 121 of main question 12. The character size of the question text of the main question ranges from 16 pt to 24 pt, so we start with 24 pt. As shown in FIG. 12A, 24 pt and 20 pt do not fit in the area, but 18 pt fits in the area. The control unit 11 calculates the size of the area required by the question 111 from the character size, number of characters, character spacing, and line spacing, and compares the calculated size with the size of the large question area of the template to determine whether it will fit. As a result of the process, the question 111 is determined to be 18 pt. Next, the process moves to the question 121 of the large question 12. As with the question 111, the character size starts at 24 pt, and the size is gradually reduced. However, as shown in FIG. 12B, the question 121 does not fit in the area even with the minimum of 16 pt, so the template is not selected. The same process is performed on the templates shown in FIG. 6A, FIG. 6B, FIG. 6C, and FIG. 6D, and only the templates for which it is determined that all the large questions, small questions, and answer columns fit into the corresponding areas remain as the selection results of step S22.

When executing step S23, the control unit 11 compares the aspect ratio of the figure to be placed with the aspect ratio of the figure area of each template to calculate how much the aspect ratio needs to be changed in order to place it. FIG. 13 is an explanatory diagram illustrating the process of determining whether or not a figure can be placed. In FIG. 13, it is determined whether or not the graph G can be placed in area A1 or area A2. Area A1 is the figure area of the template shown in FIG. 6A. Area A2 is the figure area of the template shown in FIG. 6D. The allowable aspect ratio difference set in the user setting DB 134 is ±1. If the aspect ratio of the graph G is 3:4, the allowable aspect ratio range is from 2:3 to 4:5. The aspect ratio of area A1 is 3:4.5, which is within the allowable range. On the other hand, the aspect ratio of area A2 is 4:3, which is outside the allowable range. Here, the template shown in FIG. 6A is selected, and the template shown in FIG. 6D is not selected. Similar processing is performed on the templates selected up to step S22. The template selected in step S23 also becomes the selected template as a result of the template selection processing.

If multiple templates are selected as a result of the template selection process, the control unit 11 uses each selected template in step S4 of the generation process to arrange main questions, sub-questions, answer columns, and diagrams, and generates an image of the arrangement results. In step S5, the control unit 11 calculates the blank space ratio of each generated image. For example, the blank space ratio is calculated using the following formula.

Margin ratio = area of margin area / total area of generated image … (formula)

White space is an area other than text and images. Figure 14 is an explanatory diagram showing an example of a generated image. In Figure 14, the question text is shown as a line segment, but in reality the question formula or question text is displayed. Figure 14A is an example of an image generated using the template shown in Figure 6A, Figure 14B is the template shown in Figure 6C, and Figure 14C is the template shown in Figure 6B. When the white space ratios of the three images are calculated, the image in Figure 14A is low, the image in Figure 14B is in the middle, and the image in Figure 14C is high. Here, if the number of images to be finally outputted, n = 2, the images in Figures 14A and 14B are output, and the image in Figure 14C is discarded.

In the explanation so far, the template shown in Figure 6 has been used as an example. The template shown in Figure 6 has a structure in which the question text and answer column of the main question and sub-question are close to each other, but this is not limited to this. A template with a structure in which the question text and answer column are separated may also be used. Figure 15 is an explanatory diagram showing an example of a layout template. Figure 15A is a reprint of Figure 6A. If the template shown in Figure 15A is changed to a structure in which the question text and answer column are separated, the template will look like Figure 15B.

The above explanation assumes that the subject is one in which questions are written horizontally, such as arithmetic, but this is not limited to this. Teaching material images can also be generated using a similar process for subjects in which questions are written vertically, such as Japanese. Figure 16 is an explanatory diagram showing an example of a placement template. The template shown in Figure 16 is an example in which the questions and answer columns are written vertically. Figure 16A is an example of a template with a structure in which the question text and answer columns for main and sub questions are close to each other, and Figure 16B is an example of a template with a structure in which the question text and answer columns are separated. The templates shown in Figures 16A and 16B do not have a placement area for figures and tables, but a placement area for figures and tables may be provided.

This embodiment has the following advantages. The areas for the questions and answer columns are arranged in a layout suitable for digital devices, reducing unnecessary blank space and improving visibility. Since the questions and answer columns do not need to be arranged manually, it is possible to reduce the workload associated with generating teaching material images. In addition, it is possible to make modifications such as replacing or not displaying parts of a question, and to generate teaching material images in which the questions and answer columns are arranged in separate areas, making it possible to generate a variety of teaching material images.

(Embodiment 2)
In the first embodiment, question data stored in advance in the main question DB 131 and the sub-question DB 132 is used, but in the present embodiment, image data obtained by scanning paper teaching materials is used. In the present embodiment, the hardware configuration of the generation server 1 is substantially the same as in the first embodiment, so a detailed description is omitted. The hardware configuration of the user terminal 2 is the same as in the first embodiment, so a description is omitted.

The process in this embodiment is outlined below. 1. The control unit 11 recognizes the position of each area (main question, sub-question, answer column, diagram) and the text contained in the area. 2. The control unit 11 associates each area. For example, there is a one-to-one relationship between a sub-question and an answer column, but there is usually a one-to-many relationship between a main question, sub-questions, and answer columns. 3. The control unit 11 estimates the order of each area and extracts areas that correspond to each other. 4. The control unit 11 generates a teaching material image based on the questions and answer columns to be placed.

Figure 17 is an explanatory diagram showing an example of an input image. The input image includes two main questions, main question 17 and main question 18. Main question 17 includes a question statement 171, four sub-questions 173, and four corresponding answer columns 174. Main question 18 includes a question statement 181, four sub-questions 182, and four corresponding answer columns 183, as well as a graph 184. In the following explanation, the processing performed by the generation server 1 will be explained using the teaching material image shown in Figure 17 as an example of an input image.

When a teaching material image is input, the control unit 11 recognizes the positions and sizes of the main question, sub-question, answer column, and diagram areas. Area recognition is performed using, for example, R-CNN (Regions with Convolutional Neural Networks). However, any algorithm that can detect objects contained in an image can be used. Figure 18 is an explanatory diagram showing the results of area recognition. The main question, sub-question, answer column, and diagram areas have been recognized.

Once the control unit 11 has recognized the areas, it performs character recognition for the main question, sub-question, and answer column areas using OCR (Optical Character Recognition) technology to create text data. The control unit 11 may also perform character recognition using CNN. Figure 19 is an explanatory diagram showing an example of the results of OCR. The control unit 11 recognizes the characters contained in the main question, sub-question, and answer column.

Next, the control unit 11 performs area matching. For example, if a question number is assigned to the sub-question and the answer column also contains a question number, the correspondence between the sub-question and the answer column can be understood by the question number. If there is no question number, the control unit 11 matches those that are close in distance. For example, the control unit 11 calculates the coordinate distance between the center points of the sub-question and the answer column. The control unit 11 matches those with a small difference in distance in the y direction. If there are multiple candidates to match, the control unit 11 matches those with a small difference in distance in the x direction. Note that if the input image is long in the landscape direction, the control unit 11 first extracts candidates based on the distance in the x direction, and then narrows down the candidates based on the distance in the y direction. Next, the control unit 11 detects the area between the main question and the next main question, or the area between the main question and the bottom of the image, and matches all the sub-questions and figures and charts included in the detected area with the main question. Figure 20 is an explanatory diagram showing the correspondence between areas.

Finally, the control unit 11 estimates the order of the sub-questions. If question numbers are present, the control unit 11 determines the order based on the question numbers extracted by OCR. If question numbers are not present, the control unit 11 assigns numbers in order of distance in the y direction (x direction if the image is horizontal). The control unit 11 sorts the sub-questions or answer columns in ascending order by the center coordinates in the y direction (x direction), and the order of the sorted results becomes the order. If there are multiple questions with close center coordinates, the control unit 11 sorts them in ascending order by the center coordinates in the x direction (y direction), and the order of the sorted results becomes the order.

Based on the above results, the control unit 11 obtains the data and correspondence of the main question, sub-questions, answer columns, and diagrams as shown in FIG. 11. The control unit 11 stores the data of the main question, sub-questions, answer columns, and diagrams in the main question DB 131 and sub-question DB 132. Thereafter, the control unit 11 generates a teaching material image by performing the process described in the first embodiment.

The above series of processes can be expressed as a flow chart as follows. FIG. 21 is a flow chart showing an example of the procedure for the question data creation process. The control unit 11 acquires image data by scanning paper teaching materials (step S31). The control unit 11 recognizes the areas of main questions, sub-questions, answer columns, and diagrams, and recognizes and extracts the text contained in each area (step S32). The control unit 11 estimates the correspondence between the areas (step S33). At this time, the control unit 11 uses the recognized text, such as the question number, as necessary. The control unit 11 links the areas with the order (step S34). At this time, the control unit 11 determines the order using the recognized text, such as the question number, as necessary. The control unit 11 stores the results in the main question DB 131 and the sub-question DB 132 (step S35).

In addition to the effects of embodiment 1, this embodiment has the following effects: Since structured data is created by processing image data obtained by scanning paper teaching materials, it is possible to create teaching material images suitable for display on a computer screen from paper teaching materials.

In the above explanation, it is assumed that the layout template is prepared in advance by a person, but the layout template may be generated by a learning model. FIG. 22 is an explanatory diagram showing an example of a learning model. The learning model M includes a network N1 and a network N2. The network N1 is a multi-modal embedding network. The network N2 is a layout generative network. The network N1 accepts problem data (problem text, answer column), figure image, text ratio, and image ratio, and outputs a feature vector. The text ratio is the area ratio of the text to the whole, and the image ratio is the area ratio of the image to the whole. The network N2 outputs a layout template image or layout information from random noise and the feature vector output by the network N1. The layout information is numerical information including position coordinates (placement location) and layout size indicating the layout of the problem and figure.

The learning model M is trained by preparing training data consisting of problem data, diagram images, text ratio and image ratio, and a layout template image or layout information suitable for these. The problem data, diagram images, text ratio and image ratio, and random noise are input to the learning model M, and the parameters of the learning model M are adjusted so that the layout template image or layout information output by the learning model M is close to the correct layout template image or layout information. Note that when the learning model M is in operation, the learning model M may be retrained using the adopted layout template image or layout information. The learning model M is not limited to the models shown above, and may be other models such as LayoutGAN.

It is also possible to arrange the image data of main questions, sub-questions, answer columns, and diagrams according to a predetermined rule or algorithm without using a placement template. For example, if the area occupying each element is considered as a tile, a solution algorithm for a problem called a floorplanning problem or tiling problem is used. A floorplanning problem or tiling problem is a problem in which, when a two-dimensional placement area and a large number of rectangular tiles of various sizes are given, the problem is to consider a placement of tiles that will fill the placement area as much as possible. This solution algorithm makes it possible to arrange the image data of main questions, sub-questions, answer columns, and diagrams.

Figure 23 is a block diagram showing an example of the functional configuration of the generation server. The generation server 1 includes an acquisition unit 11a, a generation unit 11b, an output unit 11c, an identification unit 11d, and a storage unit 11e. When the control unit 11 executes the control program 1P, the generation server 1 operates as follows.

The acquisition unit 11a acquires multiple pieces of question data that associate questions with answer columns. The generation unit 11b generates a teaching material image in which each piece of question data is arranged according to the page size and the layout size of the question data. The output unit 11c outputs the generated teaching material image. The identification unit 11d identifies question order information from the result of sorting the question number information in ascending order. The question data is associated with multiple groups, and the memory unit 11e stores the attributes of each group. The generation unit 11b has a selection unit 11f, which selects an appropriate layout template based on the layout template and the attributes.

The technical features (constituent elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed.
The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims.

100 Teaching material generation system 1 Generation server 11 Control unit 11a Acquisition unit 11b Generation unit 11c Output unit 11d Identification unit 11e Storage unit 11f Selection unit 12 Main storage unit 13 Auxiliary storage unit 131 Big question DB
132 Small Questions DB
133 Template DB
134 User setting DB
14 Communication unit 15 Reading unit 1P Control program 1a Portable storage medium 1b Semiconductor memory 2 User terminal B Bus N Network

Claims (14)

An acquisition unit that acquires a plurality of pieces of question data each of which associates a question with an answer column;
a generating unit that generates a teaching material image in which each of the question data is arranged according to a page size and an arrangement size of the question data;
an output unit that outputs the generated teaching material image ;
The question data is associated with each of a plurality of groups, and a storage unit stores attributes of each group.
Equipped with
The attributes include the number of questions and answers included in the question data, the number of text characters, the image size, the number of images, association information between the questions and the answers or figures, number information of the questions and answers, and order information of the questions and answers.
23. An information processing apparatus comprising: The information processing apparatus according to claim 1 , wherein the question data includes image and text data. The information processing device according to claim 1, further comprising an identification unit that identifies the number information of the questions and answers from a number indicating a number contained in the text data of the questions or images, and that identifies the order information of the questions from a result of sorting the number information of the questions in ascending order. The information processing apparatus according to claim 1 , wherein the generating unit has a selection unit that selects a placement template based on the attribute, and determines a placement location based on the placement template to generate the teaching material image. The information processing apparatus according to claim 4 , wherein the placement template includes information for specifying placement areas of questions, answers, and figures for each attribute. The information processing device according to claim 5 , wherein the selection unit selects the placement template in which the number of questions and answers in the question data to be placed matches the number of placement areas for the questions and answers on the placement template. 7. The information processing apparatus according to claim 5 , wherein the selection unit selects the placement template having an area in which all text included in the question data to be placed can be placed under a predetermined character size and font. The information processing device according to claim 5 , wherein the selection unit selects the placement template having an area in which all images included in the problem data to be placed can be placed under a condition of a predetermined aspect ratio. the selection unit selects the placement template corresponding to the page size;
The information processing apparatus according to claim 5 , wherein the generating unit generates the teaching material image having a size conforming to the page size. The information processing device according to claim 5 , wherein the placement template has a structure in which placement areas for questions and answers are the same, or a structure in which placement areas for questions and answers are divided. The information processing apparatus according to claim 1 , wherein the generating unit determines a placement location based on a rule that sets a placement position and a placement size, and generates the teaching material image. The generating unit generates a plurality of the teaching material images,
The information processing apparatus according to claim 1 , wherein the output unit displays a comparison of the generated teaching material images. The computer
Obtaining multiple pieces of question data that associate questions with answer columns;
generating a teaching material image in which each question data is arranged according to a page size and an arrangement size of the question data;
Outputting the generated teaching material image;
The question data is associated with each of a plurality of groups, and attributes of each group are stored;
The attributes include the number of questions and answers included in the question data, the number of text characters, the image size, the number of images, association information between the questions and the answers or figures, number information of the questions and answers, and order information of the questions and answers.
23. An information processing method comprising: On the computer,
Obtaining multiple pieces of question data that associate questions with answer columns;
generating a teaching material image in which each question data is arranged according to a page size and an arrangement size of the question data;
Outputting the generated teaching material image;
The problem data is associated with each of a plurality of groups, and attributes of each group are stored.
Execute the process ,
The attributes include the number of questions and answers included in the question data, the number of text characters, the image size, the number of images, association information between the questions and the answers or figures, number information of the questions and answers, and order information of the questions and answers.
A computer program comprising:

Images