JP7795660B2 - Content generation system, method, program, and storage medium - Google Patents

Description

The present invention relates to a content generation system, method, program, and storage medium.

A method for controlling a virtual character (or "avatar") using a multimodal model is known (see Patent Document 1).

Special Publication No. 2022-534708

However, no previous proposals have been made regarding generating content consisting of frames containing characters with expressions intended by the user.

The problem that this invention aims to solve is to easily generate content made up of frames containing characters with facial expressions that the user intends.

[1] A content generation system according to one aspect includes:
A content generation system that generates content consisting of one or more frames including moving or still images of a character,
a display control unit that displays, on a display unit, a user interface including a first two-dimensional operation area and a preview image of a frame to be edited by a user;
an expression determination unit that determines an expression of the character on the preview image in accordance with the position of a pointer on the first two-dimensional operation area;
a content generation unit that generates content consisting of one or more frames including the frame edited by the user;
Equipped with.

[2] A content generation system according to one aspect is the content generation system according to [1] above,
The user interface includes a second two-dimensional operation area, and further includes a line of sight determination unit that determines the line of sight of the character on the preview image according to the position of a pointer on the second two-dimensional operation area.

[3] A content generation system according to one aspect is the content generation system according to [1] or [2] above,
The game device further includes a motion determination unit that determines a motion of the character on the preview image in response to an operation of the user interface by the user.

[4] A content generation system according to one aspect is the content generation system according to [3] above,
the user interface includes a one-dimensional operation area;
The motion determination unit determines a reproduction range of the motion according to the position of the knob on the one-dimensional operation area.

[5] A content creation system according to one aspect is the content creation system according to any one of [1] to [4] above,
The device further includes a motion determination unit that determines a motion of the character on the preview image in accordance with the position of the pointer on the first two-dimensional operation area.

[6] A content creation system according to one aspect is the content creation system according to any one of [1] to [5] above,
The image display device further includes a character determination unit that determines a character to be displayed on the preview image.

[7] A content creation system according to one aspect is the content creation system according to any one of [1] to [6] above,
The image processing device further includes a background image determination unit that determines a background image for the preview image.

[8] A content creation system according to one aspect is the content creation system according to any one of [1] to [7] above,
The display device further includes a text information determination unit that determines at least one of the text to be displayed in the preview image, the type of speech bubble in which the text is displayed, the color of the text, and the size of the text in response to an operation of the user interface by the user.

[9] A content creation system according to one aspect is the content creation system according to any one of [1] to [8] above,
The first two-dimensional operation area is a user interface for setting the emotion of the character.

[10] A content creation system according to one aspect is the content creation system according to any one of [1] to [9] above,
The system further includes a voice generation unit that generates voice data for a character by inputting image data of the character selected by the user into a voice model that uses a character image and voice data associated with the character image as training data.

[11] A content creation system according to one aspect is the content creation system according to any one of [1] to [10] above,
The system further includes a speech data generation unit that generates speech data for a character that responds to the user's speech data by inputting the user's speech data into a language model trained using the character's image data and/or voice data as part of the training data.

[12] A content creation system according to one aspect is the content creation system according to any one of [1] to [11] above,
The image processing device further includes a character image generation unit that generates a character image by inputting the image data input by the user into a trained image processing model.

[13] A content creation system according to one aspect is the content creation system according to any one of [1] to [12] above,
The image processing device further includes a background image generation unit that generates a background image by inputting the image data input by the user into a trained image processing model.

[14] A content creation system according to one aspect is the content creation system according to any one of [1] to [13] above,
The image data generated by the character image generation unit and the background image generation unit is three-dimensional image data and is placed in the metaverse space.

[15] A content creation system according to one aspect is the content creation system according to any one of [1] to [14] above,
The device further includes an NFT issuing unit that issues an NFT related to the content generated by the content generating unit and/or image data of a character within the content.

[16] A method according to one embodiment includes:
A method for generating content consisting of one or more frames including a moving or still image of a character, comprising:
a display control step of displaying on a display unit a user interface including a first two-dimensional operation area and a preview image of a frame to be edited by the user;
an expression determination step of determining an expression of the character on the preview image in accordance with the position of a pointer on the first two-dimensional operation area;
a content generation step of generating content consisting of one or more frames including the frame edited by the user;
It has.

[17] A program according to one embodiment includes:
A program for causing a computer to execute a method for generating content consisting of one or more frames including a moving image or a still image of a character, the method comprising:
a display control step of displaying on a display unit a user interface including a first two-dimensional operation area and a preview image of a frame to be edited by the user;
an expression determination step of determining an expression of the character on the preview image in accordance with the position of a pointer on the first two-dimensional operation area;
a content generation step of generating content consisting of one or more frames including the frame edited by the user;
It has.

[18] A storage medium according to one aspect includes:
A computer-readable storage medium storing the program described in [17] above.

[19] A content generation system according to one aspect includes:
A content generation system that generates content consisting of one or more frames including moving or still images of a character,
a display control unit that displays a user interface and a preview image of a frame being edited by a user on a display unit;
a character determination unit that determines a character to be displayed on the preview image in response to an operation of the user interface by the user;
an expression determination unit that determines an expression of the character in response to an operation of the user interface by the user;
a line of sight determination unit that determines a line of sight of the character in response to an operation of the user interface by the user;
a motion determination unit that determines a motion of the character in response to an operation of the user interface by the user;
a background determination unit that determines a background image of the preview image in response to an operation of the user interface by the user;
a text information determination unit that determines at least one of text to be displayed in the preview image, a type of speech bubble displaying the text, a color of the text, and a size of the text in response to an operation of the user interface by the user;
a content generation unit that generates content consisting of one or more frames including the frame edited by the user;
Equipped with.

This invention makes it easy to generate content made up of frames containing characters with expressions intended by the user.

1 is a diagram illustrating an example of a system configuration of a content generation system 1 according to an embodiment of the present invention. 1 is a block diagram showing an example of a hardware configuration of a content generation system 1 according to an embodiment of the present invention. 1 is a block diagram showing an example of a functional configuration of a content generation system 1 according to an embodiment of the present invention. 10 is a flowchart showing an example of the operation of the content generation system 1 according to the present embodiment. FIG. 10 is a diagram showing an example of a content creation screen according to the embodiment. FIG. 10 is a diagram showing an example of a content creation screen according to the present embodiment, showing a state after a character has been determined. FIG. 10 is a diagram showing an example of a content creation screen according to the present embodiment, showing a state after a character's facial expression has been determined. FIG. 10 is a diagram showing an example of a content creation screen according to the present embodiment, showing a state after a character's line of sight has been determined. FIG. 10 is a diagram showing an example of a content creation screen according to the present embodiment, showing a state in which a character motion is being selected. FIG. 10 is a diagram showing an example of a content creation screen according to the present embodiment, showing a state after a character's motion has been determined. FIG. 10 is a diagram showing an example of a content creation screen according to the present embodiment, showing a state after the placement of characters has been determined. FIG. 10 is a diagram showing an example of a content creation screen according to the present embodiment, showing a state after a background image has been determined. FIG. 10 is a diagram showing an example of a content creation screen according to the present embodiment, showing a state after text information has been determined. FIG. 4 is a diagram illustrating an example of a facial expression setting unit according to the embodiment. FIG. 4 is a diagram illustrating an example of a line-of-sight setting unit according to the present embodiment. FIG. 4 is a diagram illustrating an example of a motion setting unit according to the embodiment. FIG. 10 is a diagram illustrating another example of the facial expression setting unit according to the embodiment. 10A and 10B are diagrams illustrating another example of the line of sight setting unit according to the embodiment.

Embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that components with equivalent functions in each drawing are designated by the same reference numerals, and detailed descriptions of components with the same reference numerals will not be repeated.

(Content Generation System Overview)
The content generation system according to this embodiment is used by a user to generate content consisting of one or more frames of a manga, anime, etc. The content generation system according to this embodiment will be described in detail below.

(Configuration of content generation system)
1 is a diagram showing a schematic configuration of a content creation system 1 according to this embodiment. As shown in FIG. 1, the content creation system 1 according to this embodiment includes a user terminal 2 (terminal device) used by a user, and a server 3.

The user terminal 2 and server 3 are connected to each other so that they can communicate with each other via a network 4 such as the Internet. The network 4 may be either a wired or wireless line, and the type and form of line are not important. At least a portion of the user terminal 2 and server 3 are realized by a computer (information processing device). The user terminal 2 is, for example, a terminal device such as a personal computer, smartphone, or tablet terminal. There are assumed to be a large number of user terminals 2, depending on the number of users using the content generation system 1.

(Hardware configuration)
Next, a description will be given of the hardware configuration of the content generation system 1 according to this embodiment. Fig. 2 is a block diagram showing an example of the hardware configuration of the user terminal 2 and the server 3 included in the content generation system 1 according to this embodiment.

In the user terminal 2, the CPU 201 is a processing device that controls the overall operation of the user terminal 2. The ROM 202 is non-volatile memory that stores control programs executed by the CPU 201 and various data. The RAM 203 is volatile memory used as a load area and work area for programs executed by the CPU 201. The storage device 204 is a storage means for storing various types of information and may be built into the user terminal 2 itself or may have a removable storage medium. The input device 205 is a device through which a user of the user terminal 2 inputs information and may be, for example, a keyboard, mouse, touch panel, or microphone. The display 206 is a display device that displays various types of information (user interface, etc.). The image sensor 207 is a photoelectric conversion element that captures an image of a subject. The communication I/F (interface) 208 is an interface for connecting to the network 4. The bus 209 is a bus line that interconnects the above components.

In server 3, CPU 301 is a processing device that controls the overall operation of server 3. ROM 302 is non-volatile memory that stores control programs executed by CPU 301 and various data. RAM 303 is volatile memory used as a load area and work area for programs executed by CPU 301. Storage device 304 is a storage means for storing various information and may be built into the server 3 itself or may have a removable storage medium. Communication I/F (interface) 305 is an interface for connecting to network 4. Bus 306 is a bus line that interconnects the above components.

(Functional configuration)
Next, a functional configuration of the content generation system 1 according to this embodiment will be described. Fig. 3 is a diagram showing an example of the functional configuration of the content generation system 1 according to this embodiment.

First, the functional configuration of the user terminal 2 will be described. As shown in FIG. 3, the user terminal 2 has a communication unit 21, a control unit 22 that controls the overall operation of the user terminal 2, an input unit 23 through which the user inputs various information, an output unit 24 that outputs various information, an imaging unit 25 that captures an image of a subject, and a storage unit 26 that stores various information.

The communication unit 21 is a communication interface between the user terminal 2 and the network 4. The communication unit 21 sends and receives information between the user terminal 2 and the server 3 via the network 4.

The control unit 22 has a display control unit 22a, a frame determination unit 22b, a character determination unit 22c, a facial expression determination unit 22d, a line of sight determination unit 22e, a motion determination unit 22f, a character placement determination unit 22g, a background image determination unit 22h, and a text information determination unit 22i.

The display control unit 22a displays a content creation screen (operation screen) of the content generation app (program) related to the content generation system 1 on the output unit 24 (display unit) described below.

Figure 5 is a diagram showing an example of a content creation screen 50 (operation screen) displayed by the display control unit 22a on the output unit 24 (display unit). As shown in Figure 5, the content creation screen 50 displays multiple GUIs (Graphical User Interfaces) for editing the frames that make up the content. Specifically, the content creation screen 50 includes a tab 51 for switching editing items, a frame list area 52 that displays a list of thumbnail images of the frames that make up the content and allows the user to select a frame to edit, a preview display area 53 that displays a preview image of the frame being edited, a background image area 53a that is at least a portion of the preview display area and displays a background image, a character selection area 54 for selecting a character, a character facial expression setting area 55, a character line of sight setting area 56, a character motion setting area 57, and a cursor C that the user uses to select each GUI.

Returning to Figure 3, the frame determination unit 22b determines which frame the user will edit from one or more frames that make up the content. Specifically, the frame determination unit 22b determines which frame the user will edit in response to the user's operation of using cursor C to select a thumbnail image of a frame displayed in the frame list area 52 (see Figure 5). Figure 5 shows a state in which frame 1 has been selected by the user from the thumbnail images of frames 1 to 4 displayed in the frame list area, and frame 1 has been determined by the frame determination unit 22b to be the frame to be edited.

Note that a frame can be added to the frame list area 52 by selecting the "+" button 52a displayed within the frame list area 52. Furthermore, the selected frame (frame 1 in the example shown in FIG. 5) can be deleted from the frame list area 52 by selecting the "-" button 52b. Furthermore, by selecting the play button 52c, content made up of one or more frames (frames 1 to 4 in the example shown in FIG. 5) displayed in the frame list area 52 is generated by the content generation unit 32c (described below), and the generated content is displayed in the preview display area. For example, in the example shown in FIG. 5, frames 1 to 4, displayed as thumbnails in the frame list area 52, are displayed in the preview display area 53, switching continuously at predetermined time intervals, like a slideshow.

Returning to FIG. 3 , the character determination unit 22c determines a character to be displayed in the frame being edited by the user. Specifically, the character determination unit 22c determines a character selected by the user using cursor C from among the characters displayed in the character selection area 54 (see FIG. 5 ) as the character to be displayed in the frame being edited by the user. The image of the character determined by the character determination unit 22c is superimposed on the preview image of the frame displayed in the preview display area 53. As shown in the example of FIG. 5 , when the “Hide” button is selected in the character selection area 54, no characters are displayed in the preview display area 53. However, when an arbitrary character is selected from the character selection area 54, the character selection is determined by the character determination unit 22c, and as shown in the example of FIG. 6 , the image of character A selected by the user is superimposed on the preview image of the frame displayed in the preview display area 53. Character A displayed in the preview display area 53 can be moved and enlarged or reduced within the preview display area 53 using cursor C. For example, if a mouse is used as the input device 205 of the user terminal 2, character A may be moved to any position by operating cursor C with the mouse and dragging character A in the preview display area 53. Furthermore, character A may be enlarged or reduced in size by operating the mouse wheel during the drag operation.

The image of character A displayed in the preview display area 53 may be either a moving image or a still image, but if the motion determination unit described below determines that the character's motion (action) is "do nothing," the image of the character displayed in the preview display area 53 will be a still image.

The facial expression determination unit 22d determines the "facial expression" of character A displayed in the preview display area 53 in response to the user's operation on the facial expression setting unit 55.

Figure 14 is a diagram showing an example of the facial expression setting unit 55. As shown in Figure 14, the facial expression setting unit 55 includes a two-dimensional operation area 55a (first two-dimensional operation area) and a pointer 55b that can move on the two-dimensional operation area 55a. The facial expression setting unit 55 can set the facial expression (joy, anger, sadness, or happiness) of character A displayed in the preview display area 53 according to the position of the pointer 55b on the two-dimensional operation area 55a. Specifically, the more the pointer 55b is moved toward the "happy" side (the +X direction in Figure 14) on the two-dimensional operation area 55a of the facial expression setting unit 55, the more the facial expression of character A displayed in the preview display area 53 changes to one that expresses more "joy." Conversely, the more the pointer 55b is moved toward the "sad" side (the -X direction in Figure 14) on the two-dimensional operation area 55a of the facial expression setting unit 55, the more the facial expression of character A displayed in the preview display area 53 changes to one that expresses more "sadness." Furthermore, the more the pointer 55b is moved toward the "happy" side (the +Y direction in FIG. 14) in the two-dimensional operation area 55a of the expression setting unit 55, the more the expression of character A displayed in the preview display area 53 changes to one that expresses "joy." Conversely, the more the pointer 55b is moved toward the "anger" side (the -Y direction in FIG. 14) in the two-dimensional operation area 55a of the expression setting unit 55, the more the expression of character A displayed in the preview display area 53 changes to one that expresses "anger." In this way, the expression determination unit 22d determines the "expression" of character A displayed in the preview display area 53 depending on the position of pointer 55b in the two-dimensional operation area 55a of the expression setting unit 55. This allows the user of the content creation system 1 to set the "expression" of a character in content through intuitive operations.

Figure 7 shows the state after the user has determined the "expression" of character A by operating the expression setting unit 55 using cursor C. As shown in Figure 7, the pointer 55b on the two-dimensional operation area 55a of the expression setting unit 55 has been moved to the "joy" side, and therefore the expression of character A displayed in the preview display area 53 has changed to one expressing "joy."

Returning to Figure 3, the line of sight determination unit 22e determines the "line of sight" of character A displayed in the preview display area 53 in response to a user's operation on the line of sight setting unit 56.

FIG. 15 is a diagram illustrating an example of the eye line setting unit 56. As shown in FIG. 15, the eye line setting unit 56 includes a two-dimensional operation area 56a (second two-dimensional operation area) and a pointer 56b that can move within the two-dimensional operation area 56a. The eye line setting unit 56 can set the eye line (gaze direction) of character A displayed in the preview display area 53 according to the position of the pointer 56b within the two-dimensional operation area 56a. Specifically, the more the pointer 56b is moved toward the "right" side (the +X direction in FIG. 15) within the two-dimensional operation area 56a of the eye line setting unit 56, the more the eye line of character A displayed in the preview display area 53 turns to the right (the character's eyes move to the right as viewed from the screen). Conversely, the more the pointer 56b is moved toward the "left" side (the -X direction in FIG. 15) within the two-dimensional operation area 56a of the eye line setting unit 56, the more the eye line of character A displayed in the preview display area 53 turns to the left (the character's eyes move to the left as viewed from the screen). Furthermore, the more the pointer 56b is moved "up" (in the +Y direction in FIG. 15 ) in the two-dimensional operation area 56a of the eye line setting unit 56, the more the eye line of the character A displayed in the preview display area 53 is directed upward. Conversely, the more the pointer 56b is moved "down" (in the -Y direction in FIG. 14 ) in the two-dimensional operation area 56a of the eye line setting unit 56, the more the eye line of the character A displayed in the preview display area 53 is directed downward. In this way, the eye line determination unit 22e determines the "eye line" of the character A displayed in the preview display area 53 in accordance with the position of the pointer 56b in the two-dimensional operation area 56a of the eye line setting unit 56. This allows the user of the content creation system 1 to set the "eye line" of the character in the content through intuitive operations.

Figure 8 shows the state after the user has determined the "line of sight" of character A by operating the line of sight setting unit 56 with cursor C. As shown in Figure 8, the pointer 56b on the two-dimensional operation area 56a of the line of sight setting unit 56 has been moved to the "right" side, so the line of sight of character A displayed in the preview display area 53 has changed to face right (to the right as you face the screen).

Returning to Figure 3, the motion determination unit 22f determines the motion (movement) of character A displayed in the preview display area 53 in response to the user's operation on the motion setting unit 57.

Figure 16 is a diagram showing an example of the motion setting unit 57. As shown in Figure 16, the motion setting unit 57 includes a pull-down menu 571 for selecting a character's motion type, and a playback range setting unit 572 for setting the playback range of the selected motion type. The playback range setting unit 572 includes a first shaft 572a, a first knob 572b that can move on the first shaft 572a, a second shaft 572c, and a second knob 572d that can move on the second shaft 572c.

When the pull-down menu 571 is selected with cursor C (not shown in FIG. 16) in the motion setting unit 57, a list of the types of motion that can be set is displayed. Then, by selecting the desired type of motion from the displayed list, the motion of the character is determined by the motion determination unit 22f.

In addition, the shafts (first shaft 572a and second shaft 572c) of the playback range setting unit 572 represent the time axis of the preset motion, and by moving the knobs (first knob 572b and second knob 572d) on these shafts, the motion can be set within the desired playback range.

Figure 9 shows the state in which a list of motion types is displayed after selecting the pull-down menu 571 of the motion setting unit 57 with cursor C. For example, if "wave" is selected from the list of motion types shown in Figure 9, the motion determination unit 22f determines the character's motion to be "wave," and as shown in Figure 10, character A displayed in the preview display area 53 dynamically changes to show a "wave" motion (animation).

Returning to FIG. 3 , the character placement determination unit 22g determines the placement of character A displayed in the preview display area 53 in response to a user's operation on the character placement setting unit 58 (see FIG. 11 ). The character placement setting unit 58 includes, for example, setting items for "camera,""characterposition,""rotation," and "zoom out/in," as shown in FIG. 11 . The "camera" setting item allows the user to set the camera's angle of view, such as the character's "full body,""upperbody," or "head." The "character position" setting item allows the user to set the display position of the character in the preview display area 53. The "rotation" setting item allows the user to rotate character A in the preview display area 53 in the yaw or pitch direction at predetermined angular intervals (e.g., 90-degree intervals). Furthermore, the "zoom out/in" setting item allows the user to zoom in or out on character A displayed in the preview display area 53 by moving a knob on the shaft.

Returning to Figure 3, the background image determination unit 22h determines the background image to be displayed in the preview display area 53 in response to the user's operation on the background image setting unit 59 (see Figure 12).

Figure 12 shows a state in which a background image has been set by a user operation on the background image setting unit 59 and determined by the background image determination unit 22h. As shown in Figure 12, the background image setting unit 59 includes setting items such as "background," "background position," "background size," and "lighting."

In the "Background" setting, users can select from multiple pre-prepared background images (two-dimensional or three-dimensional images, still or moving images), as well as "monochrome" (no background image). By selecting the "Add" button, users can add their own background image. By selecting the "Auto-Generate" button, users can automatically generate a background image using a trained image processing model. Specifically, when the "Auto-Generate" button is selected, a background image generation unit (not shown) of the control unit 32 inputs the character selected by the user, the character's facial expression, gaze, motion, text information (described below), and the like, as input data into the trained image processing model, thereby generating a background image. This image processing model is trained using, for example, generative adversarial networks (GAN). Alternatively, a background image may be generated by inputting a still image, such as a landscape photo, prepared by the user into the image processing model as input data. This allows the automatic generation of a background that matches the user's preferences. Here, if the background image is generated as three-dimensional data (three-dimensional image), the three-dimensional data of the background image may be placed in the metaverse space together with the three-dimensional data of the character generated by the character image generation unit 32a.

The "Background Position" setting item allows you to set the position of the background image area 53a on the preview display area 53. Specifically, you can set the position of the background image area 53a on the preview display area 53 by moving the pointer on the two-dimensional operation area. The "Background Size" setting item allows you to set the size of the background image area 53a on the preview display area 53. Specifically, you can enlarge or reduce the size of the background image area 53a on the preview display area 53 by moving the knob on the shaft. The "Lighting" setting item allows you to select "Daytime," "Evening," "Nighttime," etc. from the pull-down menu, allowing you to set the depiction of the background image according to the time of day.

Returning to Figure 3, the text information determination unit 22i determines the text information to be displayed in the preview display area 53 in response to the user's operation on the text information setting unit 60 (see Figure 13).

Figure 13 shows the state in which text information is set by user operation on the text information setting unit 60 and determined by the text information determination unit 22i. As shown in Figure 13, the text information setting unit 60 includes setting items such as "Speech Bubble," "Font," "Text," "Text Color," and "Character Size." The "Speech Bubble" setting item allows the user to select from multiple pre-prepared speech bubble images, as well as "Hide" (no text information) and "Text Only" (no speech bubble image). The "Font" setting item allows the user to select the font type of the text to be displayed in the preview display area 53 from a pull-down menu. The "Text" setting item allows the user to input text to be displayed in the preview display area 53 through user input via the input unit 23 (described below). The "Text Color" setting item allows the user to set the color of the text to be displayed in the preview display area 53. The "Character Size" setting item allows the user to set the size of the text to be displayed in the preview display area 53. Once the text information is set by the text information setting unit 60, the text information is determined by the text information determination unit 22i and displayed in the preview display area 53. In the example shown in FIG. 13 , speech bubble image 61 is displayed in preview display area 53, in which the text "Hello" entered by the user in the "Text" setting item of text information setting section 60 is combined with the speech bubble image selected by the user in the "Speech Bubble" setting item of text information setting section 60. This speech bubble image 61 can be moved within preview display area 53 by a user operation (e.g., dragging) using cursor C. In addition, the size of speech bubble image 61 can be changed (enlarged or reduced) by operating (e.g., dragging) a zoom icon D displayed near speech bubble image 61. When an area outside preview display area 53 is selected with cursor C, the position and size of speech bubble image 61 in preview display area 53 are confirmed, and zoom icon D disappears.

Returning to Figure 3, the input unit 23 is an element through which the user of the user terminal 2 inputs information, such as a keyboard, mouse, touch panel, microphone, gesture input device, etc.

The output unit 24 is an interface that outputs various information (images and audio) from the user terminal 2 to the user, and is, for example, a video display device (display unit) such as an LCD display, or a speaker. When the output unit 24 is configured as a display unit, a GUI for accepting operations from the user is displayed on this display unit by the display control unit 22a.

The imaging unit 25 captures an image of a subject using a camera module including an image sensor in the user terminal 2. The image captured by the imaging unit 25 is output to the control unit 22, where it undergoes various image processing operations and is then displayed on the display unit (output unit 24) by the display control unit 22a, or transmitted to the server 3 via the network 4 by the communication unit 21.

The memory unit 26 is, for example, a data storage device such as an internal memory or an external memory (such as an SD memory card). The memory unit 26 stores various data handled by the control unit 22, various information downloaded from the server 3 via the network 4 by the communication unit 21, images captured by the imaging unit 25, and the like. Note that the memory unit 26 does not necessarily have to be provided within the user terminal 2; part or all of the memory unit 26 may be provided within another device that is communicatively connected to the user terminal 2 via the network 4.

Next, the functional configuration of server 3 will be described. As shown in FIG. 3, server 3 has a communication unit 31, a control unit 32, and a memory unit 33.

The communication unit 31 is a communication interface between the server 3 and the network 4. The communication unit 31 transmits and receives information between the server 3 and the user terminal 2 via the network 4.

The control unit 32 controls the overall operation of the server 3. The control unit 32 also has a character image generation unit 32a, a voice generation unit 32b, a content generation unit 32c, a speech data generation unit 32d, and an NFT issuance unit 32e.

The character image generation unit 32a generates image data (two-dimensional or three-dimensional image data) of a character based on an image received from the user terminal 2 (e.g., a facial image or a full-body image of the user captured using the imaging unit 25, or a character image prepared in advance). For example, the character image generation unit 32a may generate a character image by using an image generation AI. Specifically, the character image generation unit 32a may generate a character image by inputting image data input by the user via the user terminal 2 into an image processing model trained using, for example, generative adversarial networks (GAN). The generated character image may be stored in a character image DB 33a of the storage unit 33 (described later) or transmitted to the user terminal 2 via the communication unit 31 via the network 4.

The voice generation unit 32b generates voice data corresponding to the character image (two-dimensional image or three-dimensional image) generated by the character image generation unit 32a. Specifically, the voice generation unit 32b inputs the character image generated by the character image generation unit 32a into a trained voice synthesis model and outputs the voice data. This makes it possible to automatically generate voice data from the character image with a voice tone that matches the impression of the character's appearance, atmosphere, and personality. Note that the voice model is trained using, for example, a large number of character images and voice data associated with those character images (such as lines spoken by animated characters) as training data.

The voice generation unit 32b also performs text-to-speech synthesis, which generates voice based on the voice data generated by the voice generation unit 32b, using text determined by the text information determination unit 22i and text generated by the speech data generation unit 32d (described below) as the generated voice data. The voice generated by the voice generation unit 32b through text-to-speech synthesis is output as an audio file.

The content generation unit 32c generates content (manga, animation, etc.) consisting of one or more frames edited by user operation using various interfaces displayed on the display unit (output unit 24) of the user terminal 2. For example, the content generation unit 32c generates content data in response to the selection of the play button 52c in the frame list area 52 of the content creation screen 50 shown in FIG. 5. Here, content data refers to data in which one or more frames edited by the user are encapsulated into a single file, such as a video file such as MP4 or AVI, or a still image file such as a High Efficiency Image File Format (HEIF) or AV1 Image File Format (AVIF) that can store multiple images. If an audio file has been generated by the audio generation unit 32b, the audio file is also included. The generated content data is stored in the content DB 33e of the storage unit 33 (described below) and transmitted to the user terminal 2 via the communication unit 31 via the network 4.

The speech data generation unit 32d generates character speech data that responds to user speech data (voice data or text data) input via the input unit 23 of the user terminal 2. This allows the character generated by the character image generation unit 32a to function as a conversational bot. Specifically, the speech data generation unit 32d converts the user speech data input via the input unit 23 of the user terminal 2 into text data, inputs the text data into a language model (large-scale language model), and generates speech data by outputting character speech data that responds to the user speech. Here, the language model may be trained using character image data and/or voice data as part of the training data. As a result, by inputting the character image and voice data into the language model along with the user speech data, speech data that matches the impression of the character's appearance, atmosphere, and personality can be generated. The character speech data generated by the speech data generation unit 32d is transmitted to the user terminal 2 via the network 4 by the communication unit 31 and output as sound from the output unit 24 (speaker) of the user terminal 2.

The NFT issuing unit 32e issues an NFT (Non-Fungible Token) related to the content data generated by the content generating unit 32c or image data of characters within the content (data of character images (2D or 3D images) generated by the character image generating unit 32a). That is, the NFT issuing unit 32e converts the content generated by the content generating unit 32c and/or the image data of characters within the content into NFTs. Specifically, for example, when the NFT issuing unit 32e receives an instruction to convert the content data or character image data into NFTs in response to an operation on an interface (not shown) displayed on the display unit (output unit 24) of the user terminal 2, the NFT issuing unit 32e generates an NFT ID linked to the content data or character image data. The NFT issuing unit 32e then generates a code having metadata including the generated NFT ID and transmits it to a blockchain network (not shown). This allows the content data or character image data within the content to be converted into NFTs. Additionally, users will be able to buy and sell NFT content data and character image data on NFT exchanges, or exchange it for other NFT data.

The memory unit 33 is a data storage device such as an internal memory or external memory (such as an SD memory card). The memory unit 33 stores various data handled by the control unit 32, various information received by the communication unit 31 from the user terminal 2 via the network 4, and various databases (DBs). The databases include a character image DB 33a, a background image DB 33b, a speech bubble image DB 33c, and a content DB 33d.

The character image DB 33a stores image data (including 2D or 3D model data) of multiple characters prepared in advance. The character image DB 33a also stores character images generated by the character image generation unit 32a of the control unit 32. The character images stored in the character image DB 33a may be associated with multiple motion patterns. In response to a request from the user terminal 2, the character images stored in the character image DB 33a are transmitted to the user terminal 2 via the network 4 by the communication unit 31, and are displayed on the display unit (output unit 24) by the display control unit 22a of the user terminal 2.

The background image DB 33b stores image data of multiple background images prepared in advance. In response to a request from the user terminal 2, the background images stored in the background image DB 33b are transmitted to the user terminal 2 via the network 4 by the communication unit 31, and are displayed on the display unit (output unit 24) by the display control unit 22a of the user terminal 2.

The speech bubble image DB 33c stores image data of multiple speech bubble images prepared in advance. In response to a request from the user terminal 2, the speech bubble images stored in the speech bubble image DB 33c are sent to the user terminal 2 via the network 4 by the communication unit 31, and are displayed on the display unit (output unit 24) by the display control unit 22a of the user terminal 2.

The content DB 33 d stores data of the content generated by the content generation unit 32 c. In response to a request from the user terminal 2, the content data stored in the content DB 33 d is transmitted to the user terminal 2 via the network 4 by the communication unit 31, and is displayed on the display unit (output unit 24) by the display control unit 22 a of the user terminal 2.

Note that the memory unit 33 does not necessarily have to be located within the server 3; part or all of the memory unit 33 may be located within another device that is communicatively connected to the server 3 via the network 4.

(Example of operation)
Next, a description will be given of an example of the operation of the content creation system 1. Fig. 4 is a flowchart showing the operation of a user of the content creation system 1 to create content made up of one or more frames using a user interface displayed on the display unit (output unit 24) of the user terminal 2.

First, on the user terminal 2, the display control unit 22a displays a user interface on the display unit (output unit 24) (step S1).

Next, on the user terminal 2, the frame determination unit 22b determines the frame to be edited in response to user operations on the user interface displayed on the display unit (output unit 24) (step S2).

Next, on the user terminal 2, the character determination unit 22c determines a character in response to user operations on the user interface displayed on the display unit (output unit 24) (step S3).

Next, on the user terminal 2, the facial expression determination unit 22d determines the character's facial expression in response to user operations on the user interface (facial expression setting unit 55) displayed on the display unit (output unit 24) (step S4).

Next, on the user terminal 2, the gaze determination unit 22e determines the character's gaze in response to user operations on the user interface (gaze setting unit 56) displayed on the display unit (output unit 24) (step S5).

Next, in the user terminal 2, the motion determination unit 22f determines the character's motion in response to user operations on the user interface (motion setting unit 57) displayed on the display unit (output unit 24) (step S6).

Next, on the user terminal 2, the character placement determination unit 22g determines the placement of the characters in response to user operations on the user interface (character placement setting unit 58) displayed on the display unit (output unit 24) (step S7).

Next, on the user terminal 2, the background image determination unit 22h determines a background image in response to user operations on the user interface (background image setting unit 59) displayed on the display unit (output unit 24) (step S8).

Next, in the user terminal 2, the text information determination unit 22i determines text information in response to user operations on the user interface (text information setting unit 60) displayed on the display unit (output unit 24) (step S9).

Next, on the server 3, the content generation unit 32c generates content including one or more frames (step S10).

As described above, according to this embodiment, the content generation system 1 is a content generation system 1 that generates content consisting of one or more frames including moving or still images of a character, and is equipped with a user interface (facial expression setting unit 55) including a two-dimensional operation area 55a, a display control unit 22a that displays a preview image (preview display area 53) of the frame being edited by the user on the display unit (output unit 24), an expression determination unit 22d that determines the facial expression of the character in the preview image depending on the position of the pointer 55b in the two-dimensional operation area 55a, and a content generation unit 32c that generates content consisting of one or more frames including a frame edited by the user. Therefore, content consisting of frames including characters with facial expressions intended by the user can be easily generated.

In this embodiment, as an example, an example is described in which the facial expression setting unit 55 sets facial expressions corresponding to four emotions: "joy," "anger," "sadness," and "happiness." However, it is also possible to set facial expressions corresponding to four or more emotions. For example, it is possible to set facial expressions corresponding to eight emotions (Plutchik model): "anger," "fear," "expectation," "surprise," "joy," "sadness," "trust," and "disgust."

In addition, in this embodiment, an example of setting a character's motion using the pull-down menu 571 has been described, but motions corresponding to facial expressions that can be set by the facial expression setting unit 55 may be associated in advance, and the motion corresponding to the facial expression set by the facial expression setting unit 55 may be automatically set.

In addition, in this embodiment, as an example, an example has been described in which the facial expression setting unit 55 and the motion setting unit 57 are provided separately, and "facial expressions" and "motions" are set by each setting unit. However, "facial expressions" and "motions" may also be set by a single setting unit. For example, a single emotion setting unit having a two-dimensional operation area in which multiple emotions can be set as a user interface may be provided, and the "facial expressions" and "motions" of a character corresponding to emotions set by this emotion setting unit may be associated in advance, so that the "facial expressions" and "motions" corresponding to the emotions set by the emotion setting unit are automatically set. Here, the types of emotions set by the emotion setting unit may be, for example, the four emotions of "joy," "anger," "sorrow," and "pleasure," or the eight emotions (Plutchik model) of "anger," "fear," "expectation," "surprise," "joy," "sadness," "trust," and "disgust," or more types of emotions may be used.

In addition, in this embodiment, an example has been described in which the facial expression setting unit 55 has a rectangular two-dimensional operation area 55a, but the shape of the two-dimensional operation area is not limited to this. For example, as shown in FIG. 17, the facial expression setting unit 55 may have a circular two-dimensional operation area 55c, and each facial expression such as joy, anger, sadness, or happiness may be set by moving a dot-shaped pointer 55d within this circular two-dimensional operation area 55c. Alternatively, the shape of the two-dimensional operation area may be a shape other than rectangular or circular, and each facial expression may be set by moving the pointer within that shape.

In addition, in this embodiment, an example has been described in which the gaze setting unit 56 has a rectangular two-dimensional operation area 56a, but the shape of the two-dimensional operation area is not limited to this. For example, as shown in FIG. 18, the gaze setting unit 56 may have a circular two-dimensional operation area 56c, and the gaze in each direction may be set by moving a dot-shaped pointer 56d within this circular two-dimensional operation area 56c. Alternatively, the shape of the two-dimensional operation area may be a shape other than rectangular or circular, and the gaze in each direction may be set by moving the pointer within that shape.

Any part or all of the functional units described in this specification may be realized by a program. The program mentioned in this specification may be distributed by being non-temporarily recorded on a computer-readable recording medium, or may be distributed via a communication line (including wireless communication) such as the Internet, or may be distributed in a state where it is installed on any terminal.

Based on the above description, those skilled in the art may be able to conceive of additional effects and various modifications of the present invention, but the aspects of the present invention are not limited to the individual embodiments described above. Various additions, modifications, and partial deletions are possible without departing from the conceptual idea and spirit of the present invention as derived from the content defined in the claims and their equivalents.

For example, what is described in this specification as a single device (or component, the same applies hereinafter) (including what is depicted as a single device in the drawings) may be realized by multiple devices. Conversely, what is described in this specification as multiple devices (including what is depicted as multiple devices in the drawings) may be realized by a single device. Alternatively, some or all of the means and functions included in a certain device (e.g., a server) may be included in another device (e.g., a user terminal).

Furthermore, not all of the matters described in this specification are essential requirements. In particular, matters described in this specification but not in the claims can be considered optional additional matters.

Please note that the applicant is only aware of the inventions disclosed in the documents listed in the "Prior Art Documents" section of this specification, and that the present invention does not necessarily aim to solve the problems of the disclosed inventions. The problems that the present invention aims to solve should be determined by taking into consideration the entire specification. For example, if this specification states that a specific configuration will achieve a certain effect, it can also be said that the present invention solves a problem that is the reverse of that effect. However, this does not necessarily mean that such a specific configuration is an essential requirement.

1 Content generation system 2 User terminal 21 Communication unit 22 Control unit 22a Display control unit 22b Frame determination unit 22c Character determination unit 22d Facial expression determination unit 22e Line of sight determination unit 22f Motion determination unit 22g Character placement determination unit 22h Background image determination unit 22i Text information determination unit 23 Imaging unit 24 Input unit 25 Output unit (display unit)
26 Storage unit 3 Server 31 Communication unit 32 Control unit 32a Character image generation unit 32b Voice generation unit 32c Content generation unit 32d Speech data generation unit 32e NFT issuing unit 33 Storage unit 33a Character image DB
33b Motion pattern DB
33c Background image DB
33d Speech bubble DB
33e Content DB

Claims (17)

A content generation system that generates content consisting of one or more frames including moving or still images of a character,
a display control unit that displays, on a display unit, a user interface including a first two-dimensional operation area and a preview image of a frame to be edited by a user;
an expression determination unit that determines an expression of the character on the preview image in accordance with the position of a pointer on the first two-dimensional operation area;
a motion determination unit that determines a motion of the character on the preview image in accordance with a position of a pointer on the first two-dimensional operation area;
a content generation unit that generates content consisting of one or more frames including the frame edited by the user;
A content generation system comprising: The content generation system of claim 1, further comprising a motion determination unit that determines the motion of the character on the preview image in response to the user's operation of the user interface. the user interface includes a one-dimensional operation area;
The content generation system according to claim 2 , wherein the motion determination unit determines a range of the motion to be reproduced depending on a position of the knob on the one-dimensional operation area. The content generation system of claim 1, further comprising a character determination unit that determines the character to be displayed on the preview image. The content generation system of claim 1, further comprising a background image determination unit that determines a background image for the preview image. The content generation system of claim 1 further comprises a text information determination unit that determines at least one of the text to be displayed in the preview image, the type of speech bubble in which the text is displayed, the color of the text, and the size of the text, in response to the user's operation of the user interface. The content generation system of claim 1, wherein the first two-dimensional operation area is a user interface for setting the character's emotion. The content generation system of claim 1 further comprises a voice generation unit that generates voice data for a character by inputting image data of the character selected by the user into a voice model that uses a character image and voice data associated with the character image as training data. The content generation system of claim 1 further comprises an utterance data generation unit that generates character utterance data responsive to the user's utterance data by inputting the user's utterance data into a language model trained using character image data and/or voice data as part of training data. The content generation system of claim 1 further comprises a character image generation unit that generates a character image by inputting the image data input by the user into a trained image processing model. The content generation system according to claim 10 , further comprising a background image generation unit that generates a background image by inputting the image data input by the user into a trained image processing model. The content generation system according to claim 11 , wherein the image data generated by the character image generation unit and the background image generation unit is three-dimensional image data and is arranged in a metaverse space. The content generation system according to claim 10 , further comprising an NFT issuing unit that issues an NFT related to image data of the content generated by the content generation unit or a character in the content. A method for generating content consisting of one or more frames including a moving or still image of a character, comprising:
a display control step of displaying on a display unit a user interface including a first two-dimensional operation area and a preview image of a frame to be edited by the user;
an expression determination step of determining an expression of the character on the preview image in accordance with the position of a pointer on the first two-dimensional operation area;
a motion determination step of determining a motion of the character on the preview image according to a position of a pointer on the first two-dimensional operation area;
a content generation step of generating content consisting of one or more frames including the frame edited by the user;
A method comprising: A program for causing a computer to execute a method for generating content consisting of one or more frames including a moving image or a still image of a character, the method comprising:
a display control step of displaying on a display unit a user interface including a first two-dimensional operation area and a preview image of a frame to be edited by the user;
an expression determination step of determining an expression of the character on the preview image in accordance with the position of a pointer on the first two-dimensional operation area;
a motion determination step of determining a motion of the character on the preview image according to a position of a pointer on the first two-dimensional operation area;
a content generation step of generating content consisting of one or more frames including the frame edited by the user;
A program having: A computer-readable storage medium storing the program according to claim 15 . A content generation system that generates content consisting of one or more frames including moving or still images of a character,
a display control unit that displays, on a display unit, a user interface including a first two-dimensional operation area and a preview image of a frame to be edited by a user;
a character determination unit that determines a character to be displayed on the preview image in response to an operation of the user interface by the user;
an expression determination unit that determines an expression of the character in accordance with a position of a pointer on the first two-dimensional operation area;
a line of sight determination unit that determines a line of sight of the character in response to an operation of the user interface by the user;
a motion determination unit that determines a motion of the character in accordance with a position of a pointer on the first two-dimensional operation area;
a background determination unit that determines a background image of the preview image in response to an operation of the user interface by the user;
a text information determination unit that determines at least one of text to be displayed in the preview image, a type of speech bubble displaying the text, a color of the text, and a size of the text in response to an operation of the user interface by the user;
a content generation unit that generates content consisting of one or more frames including the frame edited by the user;
A content generation system comprising: