JP4446124B2 - How to create video content - Google Patents

Description

The present invention relates to a video content creation method , and more particularly to a video content creation method for creating a video content such as a memory video that can present past photographs of a person with dementia to the dementia company (viewer) together with narration. .

  There have been some studies aimed at activating them by presenting audiovisual stimuli to elderly people with dementia. For example, in the video rasp shown in Non-Patent Document 1, a character that talks to a person with dementia who is a video viewer is presented. In addition, the project CIRCA shown in Non-Patent Document 2 uses well-designed multimedia contents including old famous songs, images, and videos. From the viewpoint of stimulating the long-term memory of people with dementia, we selected a memory video as a material for audiovisual stimulation. The memory video is a slideshow video created using photos in an old album of people with dementia. Its clinical effectiveness has been experimentally shown in Non-Patent Document 3.

  However, creating a memory video is not a simple task. First, prepare an old album and select an appropriate photo from it. The photo you use must awaken the distant memory of the viewer who has dementia. Next, a picture is taken with a video camera in a tailored manner, and a pan and zoom video effect (a so-called Cambridge effect shown in Non-Patent Document 4) is applied as necessary. Narration is also given to attract more people with dementia through video.

These are usually carried out by volunteers with video editing skills and are not something that can be easily created by caregivers. In addition, the person with dementia gets tired of repeatedly viewing the memory video, and as a result, the effect that the memory video attracts the person with dementia like the original may be gradually lost.
Lund, DA, Hill, RD, Caserta, MS, and Wright, SD: Video Respite: an innovative resource for family, professional caregivers, and persons with dementia, The Gerontologist, Vol. 35, Issue 5 (1995) 683-687. Gowans, G., Campbell, J., Alm, N., Dye, R., Astell, A., and Ellis, M .: Designing a multimedia conversation aid for reminiscence therapy in dementia care environments, Extended abstracts of the 2004 conference on Human Factors and Computing Systems (2004) 825-836. Yasuda et al .: Making a photo video for people with dementia and evaluating their concentration 28th Annual Meeting of the Society for Higher Brain Dysfunction (2004) http://en.wikipedia.org/wiki/Ken_Burns

It is necessary to clarify what video effects or audio effects should be added to create an attractive memory video. Therefore, the inventors conducted the following experiments for evaluating the effectiveness of three typical effects, namely, the above-mentioned Cambridge effect, BGM, and narration. A memory video using all the above effects and a video excluding one of those effects were prepared for 3 people with dementia. Each video is approximately 20 minutes long. During the two-week experimental period, the caregiver was asked to change the video and audio effects once a day, and the dementia watched all types of videos. When people with dementia got tired of the video, they asked them to change the video to another type. For caregivers, how long a person with dementia has watched that type of video, and how concentrated they have been watching it, can measure the subjectivity of the caregiver on a scale of “1” to “5”. I was evaluated and recorded. After the experimental period, an interview was conducted to investigate the needs of caregivers.
As a result, the following findings were obtained.

  (1) Narration is very useful for engaging memories videos.

  (2) It is also essential to zoom in on the faces of familiar people in the photo.

  (3) Furthermore, it is important that narration and zoom are linked. That is, if the narration relates to a specific person in the photo, the narration must be given when the person zooms up.

  In this way, the memorized video has been confirmed to be effective in forming a stable mental state of dementia, but it is particularly important to add visual effects and narration. Furthermore, they need to be synchronized as described above.

Therefore, the main object of the present invention is to provide a novel video content creation method .

Another object of the present invention is to provide a video content creation method capable of creating video content in which elements such as a video effect element and a narration element are synchronized.

The invention of claim 1 is a method of creating video content using fade-out / fade-in visual effects for photo transitions and arranging narration and reaction time during photo playback , wherein the minimum fade-out / fade-in time, Exclude photos when the maximum video playback time is less than the sum of the narration time and reaction time, and add a photo when the minimum video playback time is greater than the sum of the maximum fade-out / fade-in time, narration time and reaction time This is a bidet content creation method .

  According to the first aspect of the present invention, for example, photographic image data is input to a computer (12) from a photographic data input device (20. Reference numerals exemplifying corresponding parts or elements in the embodiment; the same applies hereinafter) including an image scanner. To do. The computer adds meta information such as Dublin Core, Image Regions, and FOAF to the photographic image data. Similarly, the computer also annotates the narration and creates video content using the photographic image data stored in the storage means stored in the storage means (22), for example. The computer (12) For example, local restrictions on the video effect element, the narration element, and the reaction time element to be assigned to are set on the timeline. The time adjusting means in the rendering process (S5) is, for example, a computer, and calculates the total required time of the video effect element, the narration element, and the reaction time element for each photograph, and this is, for example, a desired playback time of the video content. Adjust so that

  According to the first aspect of the present invention, it is easily attractive by considering the semantic synchronization constraint between the video effect element such as the Cambridge effect on the annotated region of the photograph and the narration element particularly in the audio effect. Can produce slideshow video (video content). In addition, since the viewer's reaction time can be reliably ensured, there is no problem such that the next narration overlaps during the viewer's reaction.

The invention of claim 2 uses a fade-out / fade-in visual effect for photo transitions, uses a pan-zoom visual effect for person-to-person transitions in the same photo, and arranges narration and reaction time during photo playback To create a video content that excludes photos when the maximum video playback time is less than the sum of the minimum fade-out / fade-in time, narration time, reaction time and minimum pan-zoom time, and the maximum fade-out / fade-in time, This is a video content creation method in which a photo is added when the minimum video playback time is larger than the sum of the narration time, reaction time, and maximum pan-zoom time .

  In accordance with the present invention, it is easily attractive by considering the semantic synchronization constraints between the video effect elements such as the Cambridge effect on the annotated region of the photo, and the narration and reaction time elements. Video content can be produced.

  The above object, other objects, features, and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

  With reference to FIG. 1, a video content creation apparatus 10 of this embodiment includes a computer 12. As the computer 12, a personal computer or a workstation can be used.

  The computer 12 is provided with an internal memory 14 such as a hard disk or a RAM. In the internal memory 14, as tools for inputting meta information, for example, DublinCore (http://dublincore.org), ImageRegions (Http://www.w3.org), FOAF (http://www.foaf-project.org), Jena2 (2.1) (http://jena.sourceforge.net), etc. are set. All of these function as means for inputting, registering or giving meta information related to photographic image data.

  Here, meta information means structured information regarding data, and in the case of the embodiment, is information that structurally describes details of photographic image data to be handled. Giving such meta information is called annotation.

Dublin Core is known as a representative tool for handling meta information. In addition, a person's face and object area (hereinafter referred to as "region") in a photo is specified and used as meta information. Use Image Regions to hold. If the region is a person, add meta information of the person using FOAF. Regions are used to add effects such as zooming and panning between regions to still photos. Jena2 (2.1) is used to store the meta information in the database in the RDF format.

  Although not shown, the computer 12 includes a graphic board, processor, sound board, or processor, through which a GUI screen and other graphics (video) are displayed on the monitor 16 and a BGM (backplane) from the speaker 18. Sound such as (ground music) is output.

  Further, a photographic data input device 20 is connected to the computer 12. The photo data input device 20 includes at least one of an image scanner, a digital camera (digital camera), the Internet (Web), and the like. The image scanner scans past photographs of persons with dementia and inputs color or monochrome photographic image data. Digital cameras can input photographic image data taken in real time, and can also be used to take past photographs and input those photographic image data. The Internet can be used to input photographic image data of a past photo of a person with dementia transmitted from a remote location, and to capture photographic image data of an event related to the past of the dementia as needed. Still other types of photo data input devices may be used.

  Further, a database 22 is coupled to the computer 12 via an interface 24. In this embodiment, the database 22 uses a relational database called PostgreSQL7.4 (http://www.postgresql.org).

  Although not shown, the computer 12 naturally has input means such as a keyboard and a mouse.

  In order to create a memory video using the video content creation apparatus 10 of the embodiment of FIG. 1, the computer 12 and the like operate according to the procedure shown in FIG.

  First, in step S1, photographic image data that may be used for a memory video is input, and annotations are assigned to each photo. Specifically, the photographic image data of the past photograph of the person with dementia is mainly input to the computer 12 using the photographic data input device 20 of FIG.

  At this time, a GUI (Graphical User Interface) screen 26 shown in FIG. The GUI 26 includes a photo display editing area 28 that occupies most of the left side of the monitor screen. The photo display / editing area 28 displays a photo of the photo image data input thereto, and is used for editing work such as designating a region. The GUI 26 includes a thumbnail display area 30 formed on the right side of the monitor screen. A thumbnail image is displayed in the thumbnail display area 30 in order to select a photo to be used from photographic image data that has been input and searched as described below.

  The GUI 26 includes a first meta information input area 32 formed below the monitor screen, a second meta information input area 34 formed substantially at the center of the monitor screen, and a playback order formed at the lower right of the monitor screen. A setting area 36 is provided. In the meta information input area 32, meta information defined in Dublin Core related to all the photographs displayed in the area 28 is input. Further, the meta information input area 34 is an area that appears at the time of input in a pop-up format, and is used to input meta information when the region is a person using FOAF. The playback order setting area 36 is used to set the order in which the photos taken in the memory video are played back, and the order can be changed by drag and drop as will be described later.

  The photo input and meta information registration in step S1 will be specifically described. Referring to FIG. 4, when photographic image data of a photograph (a drawing is a line drawing in the drawing but is actually a photograph. The same applies to other drawings) as shown in FIG. The computer 12 displays the photograph (still image) represented by the image data in the photograph display editing area 28 of the GUI 26 of the monitor 16. At the same time, the computer 12 displays a thumbnail image of the photo in the thumbnail display area 30.

  Then, using a mouse or the like (not shown), if two regions, region 1 and region 2, are set in the photo, region 1 and region 2 are respectively displayed in the photo display editing area 28 as shown in FIG. The specified rectangular frames 29a and 29b are displayed. When such region 1 and region 2 are designated, meta information of each region shown in FIG. 6 is set in a format defined by Image Regions. That is, the coordinates (x11, y11) of the origin of the region 1 (upper left corner of the rectangular frame) and the diagonal coordinates (x12, y12) thereof are registered, and the height h1 and width w1 of the region 1 are also set as meta. Registered as information. Similarly, for the region 2, the origin coordinates, diagonal coordinates, height, and width are registered as x21, y21, x22, y22, h2, and w2, respectively.

  Thus, if a photograph is input in step S1 and meta information is registered, a database as shown in FIG. 7, for example, is created. In FIG. 7, a photo display editing area 28 shown in FIG. 5 in which an actual photo is displayed and a region is designated is drawn on the right. Then, there is “dc:” in the ellipse, and the data in the rectangle related to it is meta information registered in Dublin Core. For example, “dc: date” has the date “200407716 (July 16, 2004)”, “dc: title” has the name “at Disney Animal Kingdom”, and “dc: description” has the description The sentence “They are very happy.” Is registered as meta information.

  The meta information “imgReg:” in the ellipse is meta information when a region is designated by Image Regions. “ImgReg: has region” is meta information indicating that the region is set, “imgReg: Rectangle” indicates that the region is rectangular, “imgReg: regionDepict” is a description of the region, and “imgReg: “BoundingBox” includes the origin position and size (height h, width w) of the region. “ImgReg: coords” is the coordinates of the origin position and the diagonal position of the region.

  The meta information given by “foaf: gender” is the gender (“female (female)” in the example) when the region is a person, and the meta information given by “foaf: name” is the name (in the example "Haruka (Haruka)"), the meta information given by "foaf: Person" shows the relationship between the viewer (the person watching this memorable video) and the person in the photo. In the example, "grandchild" "It can be seen that it is.

  In addition, each meta information shown in FIG. 7 is shown below, respectively.

xmlns: imgReg = “http://www.w3.org/2004/02/image-regions#”
xmlns: foo = “xmlns.com/foaf/0.1”
xmlns: dc = “http://purl.org/dc/elements/1.1/”
When the photo input and meta information registration are performed in step S1 of FIG. 2, the computer 12 stores the photo and the meta information assigned thereto in the database 22. In the above description, one photo and its meta information have been described. However, every time photographic image data is input by the input device 20, meta information as shown in FIG. Photo data to which meta information is added is stored in the database 22.

  In step S1, the narration is further annotated to create a database. In other words, as described above, narration is important for a memory video, but narration is a very time-consuming task. Therefore, in this embodiment, in order to shorten the narration assignment time, a narration database including thousands of typical narration texts and corresponding voice data is constructed and registered in the database 22 (FIG. 1). did.

  As explained in detail later, typical narration text was created using simple rules based on Japanese grammar. A typical narration text was generated by giving the rule a combination of adverbs, adjectives, and several nouns. These words were used for narration annotations, and the combinations of these words were determined based on the co-occurrence probabilities. Then, using voice synthesis technology, or a professional narrator reads them out to create voice data, which is converted into a database and registered in the database 22 as well.

  In this example, in order to allow the video producer to easily select the most appropriate narration from thousands of narration data for all photos, and for the person in all photos, Since the narration data is narrowed down by associating both narration and photo annotations, the video producer can set the narration data only by selecting an appropriate one from the narrowed narration.

  FIG. 8 shows an example of the annotation given to the narration. A unique vocabulary was defined for narration annotations. In FIG. 8, na represents the namespace of this vocabulary. The narration text of “Narration1.wav” is specified by na: text. "Narration1.wav" points to several instances specified by na: keyword. The left instance in the figure shows the person associated with the narration text. It points to an instance of foaf: Person by na: referTo. On the other hand, the instance on the right shows what the person looks like and shows the word that dc: description used to create the narration text.

  Thereafter, in step S2 of FIG. 2, the photograph used at that time is searched from the photographs input in step S1, and a set of photographs to be used is selected. However, the meta information described above is used in the photo search. For example, the following can be used as search conditions for photographs using meta information.

  First, in FOAF, when the region type is a person, meta information that identifies the person is added, so use this FOAF meta information to search for "photos of a specific person" be able to. A plurality of persons can be searched simultaneously, but in this case, a photograph in which all of the searched names are shown is targeted.

  When using the Dublin Core meta information, you can search by “shooting date”. For example, in “From (first specified date) to To (second specified date)”, it is possible to search for all photos taken after the first specified date and before the second specified date. With “From (specified date)”, you can search all photos taken after the specified date. Similarly, “To (specified date)” can be used to search for all photos taken before the specified date. In addition, it is possible to search for “photos whose specific properties include specific values”. For example, photos with “Disney” in “dc: title”. However, a plurality of images can be selected or searched at the same time, but at the time of selecting a plurality of images, only photographs that satisfy the set search conditions at the same time are targeted.

  Then, when a set of photos to be actually used is selected from the searched photos, the reproduction order of a plurality of photos in the set is designated or determined. In order to set the playback order, for example, in the GUI 26 as shown in FIG. 9, it is displayed in the thumbnail display area 30, so by dragging and dropping a photo in the list image to the playback order setting area 36, You can set the playback order of selected photos. However, the specific details are omitted here.

  When a set of photos to be used in step S2 is selected, a priority is preferably given to each of the photos. This priority is a ranking to be used for the memory video. For example, as explained later, when the total number of hours of the entire memory video is limited to 20 minutes or 30 minutes, for example, when calculating the time required for video effects and narration, the number of photos must be reduced. Work out from the lowest priority photos.

  Subsequently, in step S3 of FIG. 2, BGM and narration to be played at the time of reproduction of the photograph are selected. In this embodiment, for example, an MP3 format music file is supported, and if a song to be used as a BGM is registered as a playlist, the BGM is played along the playlist. Further explanation of BGM is omitted here.

  Next, in step S4 of FIG. 2, it is determined whether or not to preview. “YES” is determined in step S4 when the selection of all the photos and the like are completed. If not, “NO” is determined, and the process returns to step S2 or step S3 to continue the operation. That is, each step S1-S4 of FIG. 2 can be executed any number of times at an arbitrary time by selecting from the menu, so that the producer needs only when he has time. You only need to do the necessary work. In any case, since the result of the previous work is stored in the database 22, in this work, first, the data up to the previous time is read from the database 22, and then the process that continues or changes the process. I do.

  When “YES” is determined in step S4, in the subsequent step S5, for example, a memory video in the RVML format is rendered by adding a Cambridge effect, BGM, and narration in accordance with a media synchronization method considering semantics. To do.

  Note that “RVML” is a kind of XML designed so that swf can be completely expressed, and all versions of swf moving images can be expressed as RVML. However, swf is basically only a column of tags excluding a file header having information such as a Flash version and a frame rate. For example, a pattern in which a figure is defined by a definition tag, the figure is arranged in a frame by an operation tag, and the current frame is drawn on the screen by a display tag corresponds to one frame, and this is repeated.

  The generated RVML can be used to create a flash movie format memory video by using KineticFusion (http://www.kinesissoftware.com), which is a flash tool, from the RVML.

  Specifically, step S5 in FIG. 2 is executed according to the procedure shown in FIG. In the first step S <b> 11, the computer 12 first starts playing a song in a playlist (not shown) as a BGM. Thereafter, the music is sequentially switched and continuously reproduced along the playlist. That is, the computer 12 reads the music data registered in the playlist, and processes the music data with a sound board or processor, so that the music in the playlist is reproduced as sound from the speaker 18 (FIG. 1).

  In the next step S13, the computer 12 sets the width of the photo display editing area 28 (FIG. 3) of the GUI 26 formed on the display screen of the monitor 16 as Sw and the height as Sh.

  In step S15, the computer 12 increments the number of photographs n (n = n + 1). In the next step S17, the nth photo listed in the reproduction order setting area 36 (FIG. 9) is displayed in the display editing area 28 in a fade-in manner. That is, the computer 12 reads the nth photo in the playback order and the meta information (annotation) associated therewith from the database 22 (FIG. 1), and displays the nth photo. However, since n = 1 at the beginning, the first photo is displayed in a fade-in manner.

  Then, in the next step S18, the computer 12 determines whether or not narration is given to the photograph. If the narration is already given and registered, the narration is reproduced in step S19. Therefore, the registered narration can be confirmed in step S19.

  However, when the narration has not been given yet, after the pause for a fixed time in step S20, in the next step S21, the computer 12 determines whether or not a region is designated for the nth photo, for example, Image Judged by whether there is meta information of Regions. If the region is designated, in the next step S23, the computer 12 increments the region number m (m + 1). In step S25, the meta information of Image Regions of the mth region is referred to. This meta information includes position data and size data of the mth region. Therefore, in the next step S27, the computer 12 uses the meta information and the previously set display area height Sh and width Sw to align the mth region with the center of the area 28. .

  As an example, if the width of the region is w1 and the height is h1, the enlargement ratio is min [Sw / w1, Sh / h1], and the frame is displayed frame by frame until it just fits in the center of the display screen 28. The photographic image is displayed on the screen 28 by moving dX on the horizontal axis and dY on the vertical axis, and enlarged by dZ.

  However, in step S27, the image of the region is not only aligned with the center of the display area, but may be aligned with other positions such as upper left and lower right, for example.

  Thereafter, after the pause in step S29, in the next step S31, the computer 12 determines whether or not there are no remaining regions. That is, since the number of regions can be known from the image region meta information, in step S31, it may be determined whether the result incremented in step S23 is equal to the number of regions.

  If there are remaining regions, in the next step S32, the computer 12 assigns video effects A-1 and A-2 indicated by identifiers RT-1, RT-2 and RT-4 shown in Table 1.

Here, the video effect will be described. Inventors, etc., how to add video effects (effects given to each photo, transitions between photos) that can be added to multiple creators (producers) with video editing experience when converting from photos to memory videos An interview was conducted to see if something could be expected. The results were the following A-1 to A-3 and B-1 to B-5.
(A) Effect A-1: A so-called Ken Burns effect called zooming up and panning on a rectangular area (hereinafter referred to as a region) including a human face in a photograph.
A-2: Instead of panning, the zoomed-up region is faded out, and the next region is faded in.
A-3: First, a color photograph is displayed in monochrome, and then gradually changed to color display.
(B) Transition B-1: Fade out the next photo while fading out the previous photo, and overlap them.
B-2: Slide in the next photo.
B-3: Dissolve the previous photo and move to the next photo.
B-4: The previous photo is page peeled (an effect that rises from the lower right corner) and transitions to the next photo.
B-5: The previous photo is rotated about the vertical axis at the center to transition to the next photo.

  Based on the above results, we created a template to render the memories video as shown in Table 1. The items listed in “Used Information” in Table 1 are the annotations to be given to the photos, and are designed as annotation ontologies. Semantic Web (this is a next-generation Web technology that improves search performance and improves usability as a framework for adding annotations. It includes "meta information (additional information that describes the content)" and "ontology (a term that describes metadata). Definition) ”is used. That is, the annotation is described in RDF (Resource Description Framework). This is because, in the future, when considering information exchange in which a photograph annotated by another person is used for a memorable video of yourself or a family, the compatibility with the Web is taken into consideration.

  Furthermore, the inventors have established a policy of using the existing vocabulary as much as possible, and in the experiment for the embodiment of FIG. 1, the date and event of shooting are standard vocabulary regarding bibliographic information. It was described in the above Dublin Core. In addition, in order to describe the information of the person in the photograph, the above-mentioned FOAF, which is a standard vocabulary for describing the information of the person, was used. Then, to describe the human region in the photo, use the above-mentioned Image Region, and the color of the photo is Exif (http://it.jeita.or.jp/document/publica/standard/exif/english/ (See jeida49e.htm). In addition, regarding the relationship with the person (subject) in the photo, the knows property is defined in FOAF, but in creating a memory video, the relationship between the person (viewer) and the subject (related party) is defined in more detail. Therefore, we used RELATIONSHIP (see http://vocab.org/relationship/) defined by extending the FOAF knows attribute. This was used to describe parent-child relationships, relative relationships, etc. The sample is shown in FIG.

  Furthermore, for the effects and transitions listed in Table 1, the ontology of the video effects used for creating the memory video is defined as shown in Table 2.

  Then, a conversion rule from photo annotation to video effect is described and used as a template. Table 3 shows a representation example of the rendering template “RT-8” in Table 1. However, a template is created in the same manner for other identifiers shown in Table 1.

  Here, it is assumed that the rendering template itself is also described as an RDF statement, and the rendering template is searched using the above-described RDQL. Note that the rendering ontology has a low necessity for interoperability (interoperability), so another unique format may be defined.

  Returning to step S32, specifically, the computer 12 includes a plurality of inter-region intervals from the region coordinate value (imgReg: coords) and the region boundary value (imgReg: boundingBox) in the meta information as shown in FIG. The X direction (horizontal direction) interval and the Y direction (vertical direction) interval are calculated. Then, it is determined whether the inter-region spacing is greater than or less than a predetermined threshold. When either the X direction (horizontal direction) interval or the Y direction (vertical direction) interval is equal to or less than the threshold value, after zooming in on the region including the face of the person in the photo to use the Cambridge effect, Pan to the next region. Conversely, when either the X direction (horizontal direction) interval or the Y direction (vertical direction) interval is equal to or greater than the threshold, the zoomed-up region is faded out and the next region is faded in instead of panning.

  Also, if the subject information indicated by foaf: person shown in FIG. 7 is a subject that is memorable to the viewer, it zooms up only to the region of that subject, and then pans or fades out and fades in according to the inter-region spacing. Perform the process.

  However, the conversion templates exemplified in Table 3 are used in order to automatically give A-1 and A-2 video effects such as identifiers RT-1, -2, and -4 in step S32.

  Then, it returns to previous step S23 and repeats steps S23-S32.

  If “NO” in the step S31, that is, if all the regions of the nth photo in the photo list column 38 have been processed, in the next step S33, the computer 12 displays the nth photo in the photo display editing area. Fade out from 28.

  Subsequently, in step S35, the computer 12 determines whether there is no remaining photo to be processed. That is, since the number of photos in the photo list shown in FIG. 9 is known in advance, in this step S35, it may be determined whether or not the result incremented in step S15 is equal to the number.

  If the photograph still remains, in the next step S36, the computer 12 causes the video effect A-3 indicated by the identifiers RT-3, RT-5, RT-6, RT-7, and RT-8 shown in Table 1. Or B-1 and B-5.

  Specifically, the computer 12 acquires meta information of a color tone of a photograph (not shown in FIG. 7) from the above-described Exif color space information among the meta information, and the color information is obtained from the previous photo. Judge whether the current photo shows a color change such as color in monochrome. If there is the color change information, the computer 12 gives the video effect that the current color photograph is first displayed in monochrome and then gradually changed to color display.

  In addition, the computer 12 acquires data of the shooting date (date) from the meta information shown in FIG. 7, and calculates the age difference between the previous photo and the current photo. Then, it is determined whether the age difference is greater than or equal to a predetermined threshold value. When the age difference from the previous photo is less than or equal to the threshold value, the computer 12 gives the video effect of fading in the next photo while fading out the previous photo and overlapping them. When the age difference from the previous photo is equal to or greater than the threshold, the computer 12 gives the video effect that the previous photo is rotated about the vertical axis at the center to transition to the current photo.

  Further, the computer 12 determines whether or not the scenario is consistent with reference to the shooting date and the event (title) in the meta information shown in FIG. For example, it may be determined whether the photos are taken in chronological order, or if they are of the same age, the order of the seasons is followed. When it is determined that the scenario is consistent, a video effect is applied in which the next photo is faded in while the previous photo is faded out, and they are overlapped.

  In addition, when the age difference of the photograph calculated as described above is below a predetermined threshold and is a turning point of the scenario, for example, you have entered a school, married, a child was born, etc. In this case, the computer 12 gives the video effect that the previous photo is rotated about the vertical axis at the center to transition to the current photo.

  However, the conversion template exemplified in Table 3 is used to automatically give the video effect in step S36.

  When it is determined in the previous step S35 that there is a remaining number of photos, the process returns to the previous step S15, and steps S15 to S36 are repeatedly executed. If “NO” in the step S35, the computer 12 stops the BGM in a step S37 and ends.

  In this way, according to the flowchart of FIG. 10, a series of video content (recollection video) is generated (rendered) by editing the photographic image data using the meta information related thereto and adding a video effect. However, the narration is applied simultaneously with or in parallel with the video effect application.

  When giving narration, the narration button 38 formed on the GUI 26 shown in FIG. 4 is operated. When the user needs to give narration to the generated slide show, the user clicks the narration button 38 with a mouse (not shown). Then, as shown in FIG. 11, the narration creation area 44 is displayed in a pop-up format in the form of overlapping the photo display editing area 28 and others in the approximate center of the GUI 26.

  In this narration creation area 44, a narration candidate text display area 46 is formed. The narration candidate text display area 46 displays the narration candidate text (sentence) when the computer 12 creates the narration candidate text based on the keyword acquired from the meta information or the keyword input by the user, as will be described later. It is an area for. An OK button 48 is arranged in association with the narration candidate text display area 46. This OK button 48 is for instructing whether or not to select a narration text sentence displayed in the narration candidate text display area 46. If the displayed narration candidate is used, it may be clicked.

  However, in the narration candidate text display area 46, not only one candidate but a plurality of candidates may be displayed at once. In this case, the user directly clicks the area 46. One or more narration texts may be selected.

  An instruction button 50 is formed below the narration candidate text display area 46. For example, when the computer 12 cannot create an appropriate narration candidate text, the instruction button 50 is clicked with the mouse when the user manually inputs an appropriate keyword. Then, the keyword input area 52 below it is validated. Therefore, the user can input a keyword using a keyboard (not shown). As shown in FIG. 11, the keyword input area 52 is preferably divided into a plurality of categories (five in this embodiment) so that a keyword can be input for each category. By looking at the input area 52, the user can confirm whether or not the intended keyword has been input. When confirming that the keyword is input as intended, the user may operate the OK button 54 on the right side of the input area 52. Thereby, the keyword input can be completed.

  To the right of the keyword input OK button 71, an end button 56 is set. The end button 72 is operated to end the narration assignment work.

  By operating the narration giving button 38 in this way, the GUI 26 enters the state shown in FIG. 11, and the narration giving operation shown in FIG. 12 can be executed in this state.

  In the first step S41 of FIG. 12, the computer 12 determines whether a photo to be narrated is selected. In order for the user to select a photo, the thumbnail image displayed in the thumbnail display area 30 may be clicked as described above. When the computer 12 determines that a photo has been selected in step S41 by an operation in the thumbnail display area 30, the computer 12 displays the photo represented by the selected thumbnail image in the photo display editing area 28 in the next step S43. . At this time, however, the GUI 26 is in a “narration giving mode” in which the narration creation area 44 overlaps with the photo display / edit area 28, so that the picture cannot be edited in this area 28.

  In the next step S45, the computer 12 acquires the meta information registered by the method described above, and in the next step S47, extracts a keyword from the meta information. In step S49, the computer 12 displays the narration candidate text in the narration candidate text display area 46 based on the keyword.

  Here, a method for generating narration candidates will be described.

  In the system 10 of this embodiment, typical narration patterns as shown in Table 4 are preset in the internal memory (not shown) of the computer 12 or the database 22 (FIG. 1). In this sense, the internal memory and / or the database 22 function as a narration pattern setting unit. Then, the narration text is automatically generated by applying the words X1-X5 shown in Table 4 based on the keywords extracted from the meta information.

  However, how to give the words X1-X5 follows Table 5 as an example.

  For example, when using the first pattern “Is it X1 (or is it)” or the second pattern “Is this X1 (or is it)” in Table 4, the word X1 contains people, things, places, and weather Apply noun 2, which is either However, there are several types of the noun 2, which are represented by {([adverb] + adjective) or (noun 1 +) + noun 2}. When this is decomposed, the words applied to the word X1 are “adjective + noun 2” (for example, “clean”), “adverb + adjective + noun 2” (for example, “very beautiful flower”), “adjective + noun 1”. Noun 2 "(for example," beautiful garden flower ")," adverb + adjective + noun 2 for noun 1 "(for example," very beautiful garden flower "), and" noun 1 noun 2 "(for example," Garden flowers ").

  When using the third pattern “I did X2 and X3 (or did you do it)” in Table 4, the noun 2 which is a person is applied to the word X2. However, the noun 2 in the case of X2 can be defined by {([adverb] + adjective) or (noun 1 +) + noun 2} as in the first pattern and the second pattern. Moreover, the noun 2 which is an action is applied to the word X3. The noun 2 at this time is defined by {([adverb] + noun 2)}. That is, noun 2 (action) with or without adverb (for example, “joyful trip” or “travel”).

  When using the fourth pattern “I did X3 with X4 (or did you do it)” in Table 4, the noun 2 that is a place is applied to the word X4. However, the noun 2 is defined by {([adverb] + adjective) or (noun 1 +) + noun 2} as in the case described above. The word X3 is as described above. For example, examples of places include “amusement park” and “department store”.

  When using the fifth pattern “Did you X3 with X5 (or did you do it?)” In Table 4, the noun 2 which is a time and / or place is applied to the word X5. However, the noun 2 is defined by {([adverb] + adjective) or (noun 1 +) + noun 2} as in the case described above. Examples of the weather include spring, summer, autumn and winter, and what month. The word X3 is as described above.

  Further, in the above description, the word given to the noun 2 is selected from the conceptual structure of the general noun and used with the appropriate granularity for use in narration. Then, adverbs, adjectives, and nouns 1 are assigned those that match the noun 2. However, for this “conformity”, although not shown, after creating a conformity table and determining a noun 2, adverbs, adjectives, and nouns 1 are adopted with reference to the conformity table.

  The words that can be defined as shown in Table 2 are applied to X1-X5 of the narration pattern in Table 1, but in the embodiment, this "word" is to be acquired from the meta information added to the photograph. It is.

  14 and 15 illustrate a thesaurus dictionary 58 and a co-occurrence dictionary 60 that function as word determination means in the embodiment. Both the thesaurus dictionary 58 and the co-occurrence dictionary 60 are a kind of data dictionary (data dictionary), and are preset in the database 22 shown in FIG. 1 or the internal memory of the computer 12 in the embodiment.

  As shown in FIG. 14, the thesaurus dictionary 58 shows notation fluctuations, related words, synonyms, similar words, etc., as a conceptually similar keyword tree structure or network structure hierarchical structure. It is summarized for each. In the example of FIG. 14, “place” and “action” are shown as categories. For example, “Kairakuen”, “Korakuen” and “Kenrokuen” are grouped under the concept of “park”, and the proper names “Disneyland” and “USJ” are all applied to the concept of “amusement park”. These “park” and “amusement park” are included in “location” as a category together with another concept such as “department store”. The same applies to the category of “action”. “Travel” is collectively expressed as “Travel”, and “Travel” is classified into the category of “Action” along with “Walk” and “Outing”. Similarly, keywords are accumulated for other categories “people”, “time points”, etc. suitable as “noun 2” in the above narration pattern.

  A relationship between specific words and a word and a connection are referred to as a co-occurrence relationship, and the co-occurrence dictionary 60 includes co-occurrence between keywords included in each category, for example, “place” and “action”, as shown in FIG. The relationship is shown. According to the co-occurrence dictionary 76, for example, the keyword “department” of a place is connected only to the keyword “outing” in the action, but “park” means “outing”, “walking” and It turns out that it has a strong connection to multiple keywords “travel”.

  Using such a tool, a narration candidate text is created in step S49 by applying a word, particularly the noun 2, to the narration pattern of Table 4.

  A specific example of a narration candidate will be described. The meta information shown in FIG. 7 includes “2004.07.16” as the date (dc: date), “in the Disney animal kingdom” as the title (dc: title), and “faof: person”. There is “rel: grandchildOf”, and “haruka” is included as a name (foaf: name).

  First, since the date is “2004.7.16”, referring to the thesaurus dictionary 58, although not shown in FIG. 14, it is understood that the category “time” is “summer”. When “Disney” included in the title “In Disney Animal Kingdom” is searched in the thesaurus dictionary 58 of FIG. 14, it is included in “Amusement Park”, and “Amusement Park” corresponds to the category of “Place”. I understand that. Further, referring to the co-occurrence dictionary 60 of FIG. 15, it can be seen that “Amusement park” has a co-occurrence relationship with three actions “outing”, “walking”, and “travel”. Furthermore, it is understood that there is “grandchild” as “person”, and the name is “Haruka”. In this way, when keywords are extracted from the meta information and searched for “words” applicable to the narration pattern from the keywords, the time is “summer”, the place is “amusement park”, and the action is “outing” , "Walk" or "Travel", you can see that the person is "Much of the grandchild" respectively.

  Accordingly, in step S49, the computer 12 generates three narration texts as shown in Table 6 as an example.

  The first candidate in Table 6 is obtained by selecting a first narration pattern and applying a word, and the second candidate is obtained by selecting a third narration pattern and applying a word. Are candidates for selecting a fifth narration pattern and applying a word.

  In step S49, such narration text is displayed as narration candidates at one time or sequentially in the narration candidate text display area 46 of FIG.

  In step S51, the computer 12 determines whether such a narration candidate text has been selected. If any narration candidate text is selected, in the next step S53, the computer 12 acquires the voice data of this narration text.

  In this embodiment, a speech model is registered in the database 22 of FIG. 1 or the internal memory of the computer 12, and when a narration text is determined, a narration speech is created by a speech synthesis method using the speech model. However, the voice data is generated by voice synthesis, for example, voice data uttered by a professional narrator is recorded in the internal memory or the database 22, and the voice data is edited to create a narration voice. Also good.

  In step S55, the computer 12 registers the narration audio data acquired in step S53 and the created narration audio data in the database 22 in association with the photograph selected in step S41.

  In this way, the narration audio data for one photo is added to the photo, but in the next step S57, it is determined whether there is still a photo to which narration is to be added. Since the button 56 (FIG. 11) is not pressed, the process returns to the previous step S41. However, if the end button 72 is pressed, “YES” is determined in the step S57, and the narration providing step S7 is ended.

  However, when the narration candidate displayed in the previous step S49 is not selected in step S51, the computer 12 determines whether or not there are further narration candidates in the next step S59, and if there is, the narration is performed in the next step S61. The candidate is updated and displayed again in the narration candidate text display area 46 (step S49), and the selection of the user is awaited.

  If it is determined in step S59 that there is no appropriate narration candidate, the computer 12 accepts a change keyword by the user. When inputting a keyword, the user operates the instruction button 50 (FIG. 11) and inputs the keyword in the keyword input area 52. At this time, the keywords to be entered are {person, thing, place, time, action} corresponding to noun 1 and noun 2 in the narration pattern (Table 4), {what} corresponding to adverbs and adjectives, respectively. To do. However, there may be categories and items that are not specified.

  Thus, after the user inputs a keyword, the computer 12 determines a “word” from the user input keyword using the thesaurus dictionary 58 or the co-occurrence dictionary 60 as described above in step S49. Then, the narration text is generated and displayed by applying the word to the narration pattern in Table 1. Thereafter, as described above, the user selects the displayed narration candidate text and determines the narration text.

  By registering the voice data of the narration text thus determined together with the photographic image data in the database 22, video content such as a video movie in the RVML format can be produced. Currently, two types of video effects are targeted. One is a fade-out / fade-in that is applied during transitions between photos. Another type is the effect of panning and zooming to draw the viewer's attention to several areas of the photo. In particular, the latter video effect must be accompanied by appropriate narration. For example, a narration that says “Your son is cute” must be played when you zoom in to the area that includes the son of the viewer. In addition to the relationship with the video effect, an appropriate pose (reaction time) needs to be inserted after the narration. That is, the viewer needs to give enough time to react to the narration. If the pause is too short, the next narration may block the viewer from trying to speak, which can be very frustrating for the viewer.

  In other words, in order to create a slideshow video that is of interest to viewers, it is necessary to add visual effects and audio effects such as BGM and narration. It is necessary to synchronize the audio effect. For example, Synchronized Multimedia Integration Language (SMIL) is well known as a standard expression format for multimedia contents on the web. This representation format is intended for describing multimedia content to which each content has already been assigned on the timeline. The problem with this embodiment is the semantic level synchronization between visual and audio effects in creating an attractive slideshow video from a set of annotated photos. Specifically, it is necessary to synchronize the narration or BGM and the timing for giving a visual effect (video effect) to a photograph, and this is called “media synchronization in consideration of semantics” here. I will decide.

  In order to achieve such “media synchronization in consideration of semantics”, in this embodiment, after all the above-described elements (narration, pose, video effect, photo and person in the photo) are arranged, the video effect element, Arrange them on the video timeline, taking into account the semantic constraints on the narration and pause (response time) elements.

  An example is shown in FIG. In FIG. 15, N1 (narration 1) and P1 (family) should be presented at the same time, and N2 (narration 2) and P2 (son) have similar restrictions. Furthermore, placing elements on the video timeline also requires other types of constraints to be considered, such as the relationship between narration and pose described above.

  Media synchronization considering semantics is a description of temporal constraints (local constraints) between elements constituting content, and can be expressed by a model of time interval logic. This model is well known as a description format of constraints on time relations, and is also incorporated in a part of the ontology concerning time in OWL-S. If a time interval logic model is used, the constraints between the elements in FIG. 15 are described as follows. Such restrictions are set in the internal memory 14 (FIG. 1) of the computer 12, for example.

{VE1 overlaps P1, VE1 meets N1, N1 during P1, N1 meets R1, R1 meets VE2, P1 overlaps VE2, VE2 overlaps P2, VE2 meets N2, N2 meets R2, N2 during P2}
{Visual effect or video effect VE1 overlaps with photo P1, video effect VE1 is immediately before narration N1, narration N1 is between photos P1, narration N1 is immediately before reaction time R1, reaction time R1 Is immediately before the video effect VE2, the photo P1 overlaps with the video effect VE2, the video effect VE2 overlaps with the photo P2, the video effect VE2 is immediately before the narration N2, and the narration N2 is immediately before the reaction time R2. Narration N2 is between photos P2}
Next, a procedure for calculating specific timing for presenting each of these elements in the memory video under the above-described restrictions will be described. For this calculation, the following conditions are assumed. In the following, the suffix i indicates that it is an element of each type of ith.
Condition 1: Fade-out / fade-in visual effects are used for photographic transitions. Also, pan and zoom visual effects are applied to the transition from person to person in the same picture.
Condition 2: The order of photos presented in the video is chronological order by default. Furthermore, panning and zooming between people in each photo is defaulted from left to right. However, the producer can change it if necessary.
Condition 3: The time required for fade-out / fade-in can be arbitrarily changed, but a fade-out / fade-in that is too fast or too slow may make the viewer unpleasant. For this purpose, the minimum and maximum times are given in advance. Hereinafter, the minimum time is set to minTfoi_i, and the maximum time is set to maxTfoi_i.
Condition 4: Tn_i represents time spent in Narration_i. The time required for the viewer's expected response or response is expressed in Tr_i seconds.
Condition 5: By changing the speed of panning (Δ pixel / second) and zooming (Δ multiple / second), the time required for panning and zooming can be controlled. However, for this effect as well, it is necessary to set the minimum and maximum times for the same reason as in Condition 3. The minimum time is expressed as minTpz_i, and the maximum time is expressed as maxTpz_i.
Condition 6: The producer can set the minimum and maximum video playback time of the memory video. These are expressed as minTpb and maxTpb, respectively. For example, the producer can specify that the video should be at least 30 minutes long and not longer than 60 minutes, etc.

  Under such conditions, the arrangement of the elements on the timeline is calculated.

  As described above with reference to FIG. 2, the photos used for the memory video and the order in which the people of each photo are displayed are determined in step S2. Therefore, the elements of the video effect for them can be automatically arranged. Further, the selection of the narration for each person in the photograph in step S3 is executed subsequent to the elements of the respective video effects. In the constraint for semantic synchronization set in this embodiment (local constraint between each element), a pause time for the viewer's response or reaction is arranged following the narration. Then, if there is a viewer's response or reaction, the next video effect element is immediately executed. The elements of the video effect overlap with the elements corresponding to each person in the photographs in the order of presentation. Applying similar constraints, the video effects element, narration element and reaction time element are placed up to the last photo.

  In this manner, the timeline of each element is designed by repeatedly executing steps S2 and S3 in FIG. 2, but in the rendering process executed in step S5 in FIG. Under conditions 1-6, the computer 12 calculates the exact timing of executing each element on the memory video timeline.

  First, if the maximum video playback time maxTpb is smaller than the sum of the minimum fade-out / fade-in time minTfoi_i, narration time Tn_i, reaction time Tr_i, and minimum pan-zoom time minTpz_i for all elements (video effects, narration, reaction), that is, If Σ (minTfoi_i + Tn_i + Tr_i + minTpz_i)> maxTpb when variable i changes from 1 to n, in step S5 of FIG. Excluded. However, the producer may manually select the exclusion target.

  If the minimum video playback time minTpb is larger than the sum of the maximum fade-out / fade-in time maxTfoi_i, narration time Tn_i, reaction time Tr_i, and maximum pan-zoom time maxTpz_i for all elements, that is, the variable i changes from 1 to n. If Σ (maxTfoi_i + Tn_i + Tr_i + maxTpz_i) <minTpb at that time, the computer 12 adds some photos in the rendering process of step S5, or the producer adds some photos manually. As the image data of the photograph to be added, the image data already captured in step S1 of FIG. 2 is used, or it is additionally captured whenever necessary.

  Then, time adjustment is performed by adding or deleting pictures until the video playback time length is shorter than the maximum playback time maxTpb and longer than the minimum playback time minTpb.

  In this way, attractive slideshow videos (video content) can be easily created by taking into account the semantic synchronization constraints between video effects such as the Cambridge effect on the annotated area of the photo and the narration in the audio effect. ) Can be produced. Specifically, a video effect element, a narration element, and a reaction time element are assigned to each photo on the timeline under predetermined conditions, and the total time required for each element is between the maximum video playback time and the minimum video playback time. By designing to enter, it is possible to ensure the synchronization between the video effect and the narration and the reaction time of the viewer.

It is a block diagram which shows the video content production apparatus of one Example of this invention. It is a flowchart which shows the whole operation | movement of FIG. 1 Example. It is an illustration figure which shows an example of GUI of FIG. 1 Example. FIG. 3 is an illustrative view showing one example of GUI display at the time of photo input and meta information registration used in step S1 of FIG. 2; FIG. 5 is an illustrative view showing segmentation in FIG. 4. It is an illustration figure which shows the meta information of the region of FIG. It is an illustration figure which illustrates the annotation of a photograph. It is an illustration figure which illustrates the annotation of a narration. FIG. 3 is an illustrative view showing one example of a GUI during a photo selection and playback order setting operation in step S2 of FIG. 2; It is a flowchart which shows in detail the rendering operation | movement of the memory | video in step S5 of FIG. It is an illustration figure which shows the narration preparation area | region in GUI of FIG. It is a flowchart which shows the narration provision operation | movement in step S7 of FIG. 2 in detail. It is an illustration figure which shows the thesaurus dictionary used for narration provision. It is an illustration figure which shows the co-occurrence dictionary used for narration provision. It is an illustration figure which shows an example of arrangement | positioning on the timeline of the element which considered semantics.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Video content creation apparatus 12 ... Computer 14 ... Internal memory 16 ... Monitor 20 ... Photo data input device 22 ... Database 26 ... GUI
28 ... Photo display edit area 32, 34 ... Meta information input area

Claims (2)

A method of creating video content that uses fade-out / fade-in visual effects for photo transitions and arranges narration and reaction time during photo playback ,
Exclude photos when the maximum video playback time is less than the sum of the minimum fade-out / fade-in time, narration time and reaction time, and
A video content creation method in which a photo is added when the minimum video playback time is larger than the sum of the maximum fade-out / fade-in time, narration time, and reaction time. How to create video content with fade-out / fade-in visual effects for photo transitions, pan-zoom visual effects for person-to-person transitions in the same photo, with narration and reaction time placed during the playback of the photo Because
Exclude photos when the maximum video playback time is less than the sum of minimum fade-out / fade-in time, narration time, reaction time and minimum pan-zoom time, and
A video content creation method in which a photo is added when the minimum video playback time is greater than the sum of the maximum fade-out / fade-in time, narration time, reaction time, and maximum pan-zoom time.

Images