JP7687765B2 - Caption display method and related device - Google Patents

Description

[Technical field]
The present application relates to the field of terminal technology, and in particular to a caption display method and related device.

With the rapid development of electronic products, electronic devices such as mobile phones, tablet computers, and smart TVs have become ubiquitous in people's lives, and video playback has also become an important application function of these electronic devices. When an electronic device plays a video, an application scenario in which captions related to the played video are displayed in the video playback window is also widespread. For example, captions synchronized with audio can be displayed in the video playback window, or captions input by a user can be displayed in the video playback window to improve video interaction.

However, in the aforementioned application scenario where the caption is displayed at the same time as the video is played, if the color and brightness of the video are close to the color of the caption, or the color and brightness of the caption are highly consistent with the color and brightness of the video at the display position of the caption, for example, if some bright-colored captions are displayed in a high-brightness scenario, or some white captions are displayed in a snowy scenario, the captions will not be fully recognizable and will not be clearly visible to the user, thus degrading the user experience.

An embodiment of the present application provides a caption display method and related device, which solves the problem of low caption recognition in the process of users watching videos, and improves the user experience.

According to a first aspect, an embodiment of the present application provides a caption display method. The method includes: an electronic device plays a first video. When the electronic device displays a first interface, the first interface includes a first picture and a first caption, and the first caption is displayed on a first region of the first picture in a floating manner by using a first mask as a background, the first region being a region in the first picture corresponding to a display position of the first caption. A difference value between a color value of the first caption and a color value of the first region is a first value. When the electronic device displays the second interface, the second interface includes a second picture and a first caption, no mask is displayed for the first caption, and the first caption is displayed in a floating manner on a second region of the second picture, the second region being a region in the second picture corresponding to a display position of the first caption. A difference value between a color value of the first caption and a color value of the second region is a second value, the second value being greater than the first value. The first picture is a picture in the first video, and the second picture is another picture in the first video.

In this embodiment of the present application, by implementing the above-mentioned caption display method, the electronic device can set a mask for the caption when the caption recognition is low, thereby increasing the caption recognition without changing the caption color.

In a possible implementation, before the electronic device displays the first picture, the method further includes: the electronic device obtains a first video file and a first caption file, where the first video file and the first caption file carry the same time information; the electronic device generates a first video frame based on the first video file, where the first video frame is used to generate the first picture; the electronic device generates a first caption frame based on the first caption file and obtains a color value and a display position of the first caption from the first caption frame, where the time information carried in the first caption frame is the same as the time information carried in the first video frame; the electronic device determines a first region based on the display position of the first caption; the electronic device generates a first mask based on the color value of the first caption or the color value of the first region. The electronic device overlays the first caption on the first mask in the first caption frame to generate a second caption frame, and combines the second caption frame with the first video frame. In this way, the electronic device may obtain a video file to be played and a caption file to be displayed, then decode the video file to obtain a video frame, and decode the caption file to obtain a caption frame. The electronic device may then extract caption color gamut information, caption position information, etc. from the caption frame, extract color gamut information at a caption display position in the video frame that corresponds to the caption based on the caption position information, and calculate a caption recognizability based on the caption color gamut information and the color gamut information at a caption display position in the video frame that corresponds to the caption. Furthermore, the electronic device may calculate a color value of a mask corresponding to the caption based on the caption recognizability to generate a caption frame with a mask frame, and then combine the video frame and the caption frame with the mask, and render the combined video frame.

In a possible implementation, before the electronic device generates the first mask based on the color values of the first caption or the color values of the first region, the method further includes: the electronic device determines that the first value is less than a first threshold. In this manner, the electronic device may further determine that the caption recognition degree is low by determining that the first value is less than the first threshold.

In a possible implementation, the electronic device's determining that the first value is less than the first threshold specifically includes: the electronic device divides the first region into N first sub-regions, where N is a positive integer. The electronic device determines that the first value is less than the first threshold based on the color value of the first caption and the color values of the N first sub-regions. In this manner, the electronic device may determine that the first value is less than the first threshold based on the color value of the first caption and the color values of the N first sub-regions.

In a possible implementation, the electronic device generating the first mask based on the color values of the first caption or the color values of the first region specifically includes: the electronic device determines the color values of the first mask based on the color values of the first caption or the color values of the N first sub-regions. The electronic device generates the first mask based on the color values of the first mask. In this manner, the electronic device may determine the color values of the first mask based on the color values of the first caption or the color values of the N first sub-regions, and further generate the first mask for the first caption.

In a possible implementation, the electronic device's determining that the first value is less than the first threshold specifically includes: the electronic device divides the first region into N first sub-regions, where N is a positive integer. The electronic device determines whether to combine the adjacent first sub-regions into a second sub-region based on a difference value between the color values of the adjacent first sub-regions. If the difference value between the color values of the adjacent first sub-regions is less than the second threshold, the electronic device combines the adjacent first sub-regions into a second sub-region. The electronic device determines that the first value is less than the first threshold based on the color value of the first caption and the color value of the second sub-region. In this way, the electronic device may combine the first sub-regions having close color values to generate a second sub-region, and further determine that the first value is less than the first threshold based on the color value of the first caption and the color value of the second sub-region.

In a possible implementation, the first region includes M second sub-regions, where M is a positive integer and is less than or equal to N, and the second sub-region includes one or more first sub-regions, and the number of first sub-regions included in each second sub-region is the same as or different from the number of first sub-regions included in another second sub-region. In this way, the electronic device may divide the first region into M second sub-regions.

In a possible implementation, the electronic device generating the first mask based on the color values of the first caption or the color values of the first region specifically includes: the electronic device sequentially calculates color values of M first sub-masks based on the color values of the first caption or the color values of the M second sub-regions. The electronic device generates M first sub-masks based on the color values of the M first sub-masks, where the M first sub-masks are combined into the first sub-mask. In this manner, the electronic device may generate M first sub-masks for the first caption.

In a possible implementation, the method further includes: when the electronic device displays the third interface, the third interface includes the third picture and the first caption, the first caption includes at least a first portion and a second portion, the second submask is displayed for the first portion, the third submask is displayed or not displayed for the second portion, and the color value of the second submask is different from the color value of the third submask. In this manner, the electronic device may display a caption corresponding to multiple submasks.

In a possible implementation, the display position of the first mask is determined based on the display position of the first caption. In this way, the display position of the first mask may overlap with the display position of the first caption.

In a possible implementation, the difference value between the color value of the first mask and the color value of the first caption is greater than the first value. In this way, the degree of caption recognition can be increased.

In a possible implementation, the display position of the first caption in the first picture and the second picture relative to the display screen of the electronic device is not fixed or is fixed, and the first caption is a segment of characters or symbols that are displayed consecutively. In this way, the first caption is a bullet comment .
The first caption may be a caption with a 3D barrage , or a caption synchronized with the audio, and the first caption is one caption rather than all captions displayed on the display screen.

In a possible implementation, before the electronic device displays the first interface, the method includes: the electronic device sets the transparency of the first mask to less than 100%. In this way, it can be ensured that the video frame corresponding to the area where the first mask is located is still visible to some extent.

In a possible implementation, before the electronic device displays the second interface, the method includes: the electronic device generates a second mask based on a color value of the first caption or a color value of the second region, and overlays the first caption on the second mask, where the color value of the second mask is a preset color value, and the transparency of the second mask is 100%. Alternatively, the electronic device skips generating the second mask. In this way, for a caption with high recognition, the electronic device may set a mask with a transparency of 100% for the caption, or may not set a mask for the caption.

According to a second aspect, an embodiment of the present application provides an electronic device. The electronic device includes one or more processors and one or more memories, the one or more memories coupled to the one or more processors, the one or more memories configured to store computer program code, the computer program code including computer instructions, and execution of the computer instructions by the one or more processors enables the electronic device to perform the method in any possible implementation of the first aspect.

According to a third aspect, an embodiment of the present application provides a computer storage medium. The computer storage medium stores a computer program, the computer program including program instructions that, when executed on an electronic device, enable the electronic device to perform a method according to any possible implementation of the first aspect.

According to a fourth aspect, an embodiment of the present application provides a computer program product. When the computer program product is executed on a computer, the computer is enabled to execute a method according to any possible implementation of the first aspect.

1 is a schematic flowchart of a caption display method according to an embodiment of the present application; 1 is a schematic flowchart of a caption display method according to an embodiment of the present application; FIG. 2 is a schematic diagram of a group of user interfaces according to an embodiment of the present application. FIG. 2 is a schematic diagram of a group of user interfaces according to an embodiment of the present application. FIG. 2 is a schematic diagram of a group of user interfaces according to an embodiment of the present application. 4 is a schematic flowchart of another caption display method according to an embodiment of the present application; 4 is a schematic flowchart of another caption display method according to an embodiment of the present application; 4 is a schematic flowchart of another caption display method according to an embodiment of the present application; 4 is a schematic flowchart of another caption display method according to an embodiment of the present application; FIG. 2 is a schematic diagram of a caption frame according to an embodiment of the present application. FIG. 2 is a schematic diagram of the principle of generating a mask corresponding to a caption according to an embodiment of the present application; FIG. 2 is a schematic diagram of a caption frame with a mask according to an embodiment of the present application; FIG. 2 is a schematic diagram of a group of user interfaces for displaying captions according to an embodiment of the present application. FIG. 2 is a schematic diagram of a group of user interfaces for displaying captions according to an embodiment of the present application. 1 is a schematic flowchart of a method for generating a mask corresponding to a caption according to an embodiment of the present application; FIG. 2 is a schematic diagram of another principle of generating a mask corresponding to a caption according to an embodiment of the present application; FIG. 13 is a schematic diagram of another caption frame with a mask according to an embodiment of the present application; FIG. 2 is a schematic diagram of a group of user interfaces for displaying captions according to an embodiment of the present application. FIG. 2 is a schematic diagram of a group of user interfaces for displaying captions according to an embodiment of the present application. 1 is a schematic diagram of a structure of an electronic device according to an embodiment of the present application. 2 is a schematic diagram of a software structure of an electronic device according to an embodiment of the present application; FIG. 2 is a schematic diagram of a structure of another electronic device according to an embodiment of the present application. FIG. 2 is a schematic diagram of a structure of another electronic device according to an embodiment of the present application.

Hereinafter, the technical solutions in the embodiments of the present application will be clearly described with reference to the accompanying drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, "/" indicates an "or" relationship. For example, A/B may represent A or B. In this specification, "and/or" is merely an associated relationship to describe related objects, and represents that three relationships may exist. For example, A and/or B may represent the following three cases: only A exists, both A and B exist, and only B exists. In addition, in the description of the embodiments of the present application, "a plurality of" means "two or more".

In the specification, claims, and accompanying drawings of this application, it should be understood that terms such as "first", "second", etc. are intended to distinguish between different objects and are not intended to indicate a particular order. In addition, terms such as "include", "have", and any other variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the recited steps or units, and optionally further includes unrecited steps or units, or optionally further includes another unique step or unit of the process, method, product, or device.

The term "embodiment" as used herein means that a particular feature, structure, or characteristic described with reference to the embodiment may be included in at least one embodiment of the present application. Expressions appearing in various places in this specification do not necessarily refer to the same embodiment, nor are they an independent or optional embodiment exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

To facilitate understanding, we first describe some relevant concepts in the embodiments of this application.

1. Video Decoding:

The process by which the binary data of a video file is read and interpreted according to the video file's compression algorithm to obtain image frames (sometimes called video frames) of data for video playback.

2. Caption:

Text and symbolic information that appears in the video playback window while a video is playing and is unrelated to the video file.

3. Video playback:

The process of displaying a group of images and corresponding audio information in chronological order in a video playback window after operations such as video decoding and video rendering have been performed on a video file.

4. Comments on the barrage:

A caption that can be entered by a user on a video playback client (also called a video application) and displayed in the video playback window of the entering user, or in the video playback window of another user on the video playback client, based on the position of the image frame for video playback that corresponds to the time entered by the user.

With the rapid development of electronic products, electronic devices such as mobile phones, tablet computers, and smart TVs have become ubiquitous in people's lives, and video playback has also become an important application function of these electronic devices. When an electronic device plays a video, an application scenario in which captions related to the played video are displayed in the video playback window is also widespread. For example, captions synchronized with audio can be displayed in the video playback window, or captions entered by a user (i.e., barrage comments) can be displayed in the video playback window to improve video interaction.

In an application scenario where captions synchronized with audio are displayed in a video playback window, usually at a lower position of the video playback window, a matching is performed between the timestamps of the captions and the timestamps of the image frames played in the video, and the captions are composited with the corresponding image frames played in the video. Specifically, the captions are overlaid on the corresponding video frames, and the overlapping positions of the captions and the video frames are fixed .

In an application scenario in which captions entered by users (i.e., barrage comments) are displayed in a video playback window, there are multiple captions in the video playback window that fly from left to right or right to left during the video playback process, and the overlapping positions of the captions and the video frames are not fixed .

In some practical application scenarios, in order to improve the enjoyment of video playback, the video playback platform usually provides the user with the ability to select the color of the caption by the user. In an application scenario in which captions synchronized with audio are displayed in the video playback window, the color of the caption is usually the system default color, and when playing the video, the user may select a preferred caption color. In this case, the electronic device displays the caption in the video playback window based on the color selected by the user. In an application scenario in which barrage comments are displayed in the video playback window, the user who sends the barrage comment may select the color of the barrage comment to be sent, and the color of the barrage comment seen by another user will match the color of the barrage comment selected by the user who sends the barrage comment. Therefore, when a user views the barrage comment, the colors of the barrage comments displayed in the same video frame may be different.

To implement the aforementioned two application scenarios, an embodiment of the present application provides a caption display method. The electronic device may first obtain a video file to be played and a caption file to be displayed in a video playback window, and then perform video decoding on the video file to obtain video frames and caption decoding on the caption file to obtain caption frames, respectively. The electronic device may then align and match the video frames and the caption frames based on a time sequence to synthesize a final video frame to be displayed, which is stored in a video frame queue. After that, the electronic device may read and render the video frames to be displayed based on a time sequence, and finally display the rendered video frames in the video playback window.

The following describes in detail the steps of the above-mentioned caption display method.

Figures 1A and 1B show an example of a method procedure for displaying captions according to one embodiment of the present application.

As shown in Figures 1A and 1B, the method may be applied to an electronic device 100 having video playback capabilities. Certain steps of the method are described in detail below.

Phase 1: Obtaining video and caption information streams

S101 and S102: The electronic device 100 detects a user's operation to play a video on a video application, and in response to this operation, the electronic device 100 may obtain a video information stream and a caption information stream.

Specifically, a video application may be installed on electronic device 100. After detecting a user's operation to play a video on the video application, in response to the operation, electronic device 100 may obtain a video information stream (or referred to as a video file) and a caption information stream (or referred to as a caption file) corresponding to the video that the user wants to play.

2A illustrates a user interface (UI) provided by electronic device 100 to display applications installed on electronic device 100. Electronic device 100 may detect an operation (e.g., a tap operation) performed by a user on a "Video" application option 211 in user interface 210. In response to this operation, electronic device 100 may display the exemplary user interface 220 shown in FIG. 2B. User interface 220 may be a main interface of the "Video" application. Electronic device 100 may detect an operation (e.g., a tap operation) performed by a user on a video playback option 221 in user interface 220, and in response to this operation, electronic device 100 may obtain a video information stream and a caption information stream corresponding to the video.

The video information stream and the caption information stream may be files that the electronic device 100 downloads from a video application server or may be files obtained from the electronic device 100. Both the video file and the caption file carry time information.

It can be appreciated that Figures 2A and 2B are merely illustrative of an exemplary user interface on electronic device 100 and do not constitute a limitation on this embodiment of the present application.

Phase 2: Video Decryption

S103: A video application on the electronic device 100 sends a video information stream to a video decoding module on the electronic device 100.

Specifically, after obtaining the video information stream, the video application may send the video information stream to a video decoding module.

S104 and S105: The video decoding module on the electronic device 100 decodes the video information stream to generate video frames and transmits the video frames to a video frame synthesis module on the electronic device 100.

Specifically, after receiving the video information stream sent by the video application, the video decoding module may decode the video information stream to generate video frames. The video frames may be all video frames in the video playback process. The video frames may also be referred to as image frames, and each video frame may carry time information (i.e., a timestamp) of the video frame. The video decoding module may then send the video frames generated through decoding to a video frame synthesis module, which may then generate the video frames to be displayed.

The video decoding module may decode the video information stream by using a video decoding method in the prior art, which is not limited in this embodiment of the present application. For a specific implementation of the video decoding method, please refer to technical documents related to video decoding. Details will not be described here.

Phase 3: Decoding captions

S106: The video application on the electronic device 100 sends the caption information stream to a caption decoding module on the electronic device 100.

Specifically, after obtaining the caption information stream, the video application may send the caption information stream to a caption decoding module.

S107 and S108: The caption decoding module on the electronic device 100 decodes the caption information stream to generate caption frames and sends the caption frames to the video frame synthesis module on the electronic device 100.

Specifically, after receiving the caption information stream sent by the video application, the caption decoding module may decode the caption information stream to generate caption frames. The caption frames may be all caption frames in the video playback process. Each caption frame may include caption text, a display position of the caption text, a font color of the caption text, a font format of the caption text, etc., and may further carry time information (i.e., a timestamp) of the caption frame. The caption decoding module may then send the caption frames generated through decoding to a video frame synthesis module, which may then generate the video frames to be displayed.

The caption decoding module may decode the caption information stream by using a conventional caption decoding method, which is not limited in this embodiment of the present application. For a specific implementation of the caption decoding method, please refer to technical documents related to caption decoding. Details will not be described here.

Please note that the example in which the video decoding step in phase 2 is performed first and then the caption decoding step in phase 3 is used simply in this embodiment of the present application. In some embodiments, the caption decoding step in phase 3 may be performed first and then the video decoding step in phase 2, or the video decoding step in phase 2 and the caption decoding step in phase 3 may be performed simultaneously. This is not limited in this embodiment of the present application.

Phase 4: Compositing, rendering, and displaying video frames

S109 and S110: The video frame synthesis module on the electronic device 100 overlays and combines the received video frame and caption frame to generate the video frame to be displayed, and sends the video frame to be displayed to a video frame queue on the electronic device 100.

Specifically, the video frame synthesis module may perform matching based on time information corresponding to the video frames and time information corresponding to the caption frames, and after the matching is completed, may overlay the caption frames onto the corresponding video frames and combine them to generate the video frames to be displayed. Then, the video frame synthesis module may send the video frames to be displayed to a video frame queue.

S111 to S113: The video rendering module may read the video frames to be displayed from the video frame queue in chronological order and render the video frames to be displayed in chronological order to generate rendered video frames.

Specifically, the video rendering module may obtain the video frames to be displayed from the video frame queue in real time (or at intervals of a period of time). After the video frame composition module sends the video frames to be displayed to the video frame queue, the video rendering module may read the video frames to be displayed from the video frame queue based on a time sequence and render the video frames to be displayed to generate rendered video frames. The video rendering module may then send the rendered video frames to the video application.

The video rendering module may render the video frames to be displayed by using a conventional video rendering method, which is not limited in this embodiment of the present application. For a specific implementation of the video rendering method, please refer to technical documents on video rendering. Details will not be described here.

S114: The electronic device 100 displays the rendered video frame.

Specifically, after receiving the rendered video frames transmitted by the video rendering module, a video application on the electronic device 100 may display the rendered video frames on a display screen (i.e., a video playback window) of the electronic device 100.

For example, Fig. 2C may be a picture of a frame in a rendered video frame displayed after the electronic device 100 executes the caption display method shown in Fig. 1A and Fig. 1B. The caption "W S Y T K L D G S Y D Z M", the caption "highly recognizable caption", and the caption "unclear color caption" are all barrage comments, and the display positions of the barrage comments are not fixed with respect to the display screen of the electronic device 100. The display position of the caption "caption synchronized with audio" is fixed with respect to the display screen of the electronic device 100. As can be easily seen from Fig. 2C, the color difference between both sides of the caption "W S Y T K L D G S Y D Z M" and the color of the video is small, so the caption recognizability is low, and as a result, the user cannot see the caption clearly. The color difference between the captions "Highly recognizable caption" and "Caption synchronized with audio" and the color of the video is large , so the caption recognition is high and users can see these captions clearly. The color difference between the caption "Obscure color caption" and the color of the video is not small, but the brightness of the video is high , so the caption recognition is also low and users cannot see this caption clearly.

As can be seen from FIG. 2C, when using the caption display method shown in FIG. 1A and FIG. 1B, in an application scenario in which captions are displayed while a video is being played, if the color of the caption overlaps significantly with the color and brightness of the video at the caption display position, the caption recognition will be low and it will be difficult for the user to clearly see the caption, resulting in a poor user experience.

In order to solve the above-mentioned problem, an embodiment of the present application provides another caption display method. The electronic device may first obtain a video file to be played and a caption file to be displayed in a video playback window, then perform video decoding on the video file to obtain a video frame, and perform caption decoding on the caption file to obtain a caption frame. The electronic device may then extract caption color gamut information, caption position information, etc. from the caption frame, and extract color gamut information at a caption display position in a video frame corresponding to the caption based on the caption position information, and then calculate a caption recognizability based on the caption color gamut information and the color gamut information at a caption display position in a video frame corresponding to the caption. If the caption recognizability is low , the electronic device may add a mask to the caption, calculate the color value and transparency of the mask based on the caption recognizability to generate a caption frame with the mask, and perform alignment and matching between the video frame and the caption frame with the mask based on a time sequence to synthesize a final video frame to be displayed. The electronic device buffers the video frames to be displayed in a video queue, then reads and renders the video frames to be displayed based on a time sequence, and finally displays the rendered video frames in a video playback window. In this way, the problem of low caption recognition can be solved by adjusting the color and transparency of the caption mask without changing the color of the caption selected by the user. In addition, this can reduce the occlusion of the video content by the caption, and ensure a certain visibility of the video content, thereby improving the user experience.

The following describes another method of displaying captions provided in one embodiment of the present application.

Figures 3A, 3B, 3C, and 3D show an example of a method procedure for another caption display method according to one embodiment of the present application.

As shown in Figures 3A, 3B, 3C, and 3D, the method may be applied to an electronic device 100 having video playback capabilities. Certain steps of the method are described in detail below.

Phase 1: Obtaining video and caption information streams

S301 and S302: The electronic device 100 detects a user's operation to play a video on a video application, and in response to this operation, the electronic device 100 may obtain a video information stream and a caption information stream.

For the specific execution process of steps S301 and S302, please refer to the relevant contents of steps S101 and S102 in the embodiment shown in Figures 1A and 1B. Details will not be described again here.

Phase 2: Video Decryption

S303: A video application on the electronic device 100 sends a video information stream to a video decoding module on the electronic device 100.

S304 and S305: The video decoding module on the electronic device 100 decodes the video information stream to generate video frames and transmits the video frames to a video frame synthesis module on the electronic device 100.

For the specific execution process of steps S303 to S305, please refer to the relevant contents of steps S103 to S105 in the embodiment shown in Figures 1A and 1B. Details will not be described again here.

Phase 3: Decoding captions

S306: The video application on the electronic device 100 sends the caption information stream to a caption decoding module on the electronic device 100.

S307: The caption decoding module on the electronic device 100 decodes the caption information stream to generate caption frames.

For the specific execution process of steps S306 and S307, please refer to the relevant contents of steps S106 and S107 in the embodiment shown in Figures 1A and 1B. Details will not be described again here.

Figure 4 shows an example of a caption frame generated by decoding the caption information stream by the caption decoding module.

As shown in FIG. 4, the area inside the rectangular solid box may represent the caption frame display area (or may be called the video playback window area) and may coincide with the video frame display area. In this area, one or more captions may be displayed, such as, for example, "W S Y T K L D G S Y D Z M", "Highly recognizable caption", "Obscure color caption", "Synchronized with audio caption". Each of "W S Y T K L D G S Y D Z M", "Highly recognizable caption", etc. may be referred to as one caption, and all the captions displayed in this area may be referred to as a caption group, and a group of captions, such as "W S Y T K L D G S Y D Z M", "Highly recognizable caption", "Obscure color caption", "Synchronized with audio caption", etc. may be referred to as a caption group.

The rectangular dashed line boxes surrounding each caption shown in FIG. 4 are merely auxiliary elements used to identify the location of each caption and may not be displayed in the video playback process.

Based on the above description and the description of captions and caption groups, as shown in FIG. 2C, it can be easily understood that the picture shown in FIG. 2C displays four captions, which are "W S Y T K L D G S Y D Z M", "Highly recognizable caption", "Obscure color caption", and "Synchronized with audio caption". These four captions form a caption group.

S308: The caption decoding module on the electronic device 100 extracts caption position information, caption color gamut information, etc. of each caption from the caption frame and generates caption group information.

Specifically, after generating the caption frame, the caption decoding module may extract caption position information, caption color gamut information, etc. of each caption from the caption frame to generate caption group information. The caption position information is the display position of each caption in the caption frame display area, and the caption color gamut information may include the color value of each caption. The caption group information may include the caption position information and caption color gamut information of all captions in the caption frame.

Optionally, the caption color gamut information may also include information such as the luminance of the caption.

The process of extracting caption position information and caption color gamut information is described in detail below.

1. The process of extracting caption location information:

The display location area of the caption can be the inner area of the rectangular dashed box shown in FIG. 4 that can exactly cover the caption, or another inner area of any shape that can cover the caption. This is not limited in this embodiment of the present application.

In this embodiment of the present application, an example is used in which the inner area of a rectangular dashed box is the display position area of the caption to explain the process of extracting caption position information.

Take the extraction of caption position information of the caption "W S Y T K L D G S Y D Z M" shown in Figure 4 as an example. The caption decoding module may first establish an X-O-Y plane rectangular coordinate system within the caption frame display area, and then select a point within the caption frame display area (e.g., the lower left corner vertex of the rectangular solid-line box) as the reference coordinate point O. The coordinates of the reference coordinate point O may be set to (0,0). As can be seen from mathematical knowledge, the coordinates (x1,y1), (x2,y2), (x3,y3), and (x4,y4) of the four vertices of the outer rectangular dashed-line box of the caption "W S Y T K L D G S Y D Z M" can be calculated. In this case, the position information of the caption "W S Y T K L D G S Y D Z M" may include the coordinates of the four vertices of the rectangular dashed box. Alternatively, since a rectangle is a regular shape, the position region of the rectangle can be determined by only determining the coordinates of the two diagonal vertices of the rectangular dashed box. Thus, the position information of the caption "W S Y T K L D G S Y D Z M" may include only the coordinates of the two diagonal vertices of the rectangular dashed box.

Similarly, the caption position information of the other captions shown in FIG. 4 can be extracted by using the caption position extraction method described above. Details will not be described again here.

Indicates that after the caption decoding module has determined the position information of all captions in the caption frame, the caption decoding module has completed extracting the caption position information.

Please note that the above-described caption location information extraction process is only a possible implementation of extracting caption location information. The implementation of extracting caption location information may alternatively be another implementation in the prior art. This is not limited in this embodiment of the present application.

2. The process of extracting caption gamut information:

First, we discuss the concepts related to the process of extracting caption gamut information below.

Color value:

A color value is a group of color values that correspond to colors in a color mode. The RGB color mode is used as an example. In the RGB color mode, colors are formed by mixing red, green, and blue, and the color value of each color may be represented by (r, g, b), where r, g, and b represent the values of the three primary colors red, green, and blue, respectively, with a value range of [0, 255]. For example, the color value of red may be represented by (255, 0, 0), the color value of green may be represented by (0, 255, 0), the color value of blue may be represented by (0, 0, 255), the color value of black may be represented by (0, 0, 0), and the color value of white may be represented by (255, 255, 255).

Color gamut:

A color gamut is a set of color values, i.e., a set of colors that can be produced in a particular color mode. It is easy to see that in the RGB color mode, a maximum of 256×256×256=16777216 different colors, i.e., 2 ²⁴ different colors, can be produced, and the color gamut is [0, 2 ²⁴ −1]. The 2 ²⁴ different colors and the color values corresponding to each color may form a color value table, in which the color values corresponding to each color may be found.

After extracting the caption position information, the caption decoding module may search a color value table for a color value corresponding to the font color based on the font color of the caption at the caption position to determine a color value of the caption.

Indicates that after the caption decoding module has determined the color values of all captions in the caption frame, the caption decoding module has completed extracting caption color gamut information.

S309: The caption decoding module on the electronic device 100 sends an instruction to the video frame gamut interpretation module on the electronic device 100 to obtain mask parameters of the caption group, which instruction carries time information of the caption frame, caption group information, etc.

Specifically, after generating the caption group information, the caption decoding module may send an instruction to the video frame gamut interpretation module to obtain mask parameters of the caption group. The instruction is used to instruct the video frame gamut interpretation module to send mask parameters (including the color value and transparency of the mask) corresponding to the caption group to the caption decoding module. The color value and transparency may be referred to as a group of mask parameters. The instruction may carry time information of the caption frame, caption group information, etc. The time information of the caption frame may be used to obtain a video frame corresponding to the caption group in a subsequent step, and the caption group information may be used to analyze the caption recognition degree in a subsequent step.

S310: The video frame gamut interpretation module on the electronic device 100 sends an instruction to the video decoding module on the electronic device 100 to obtain a video frame corresponding to the caption group, the instruction carrying time information of the caption frame, etc.

Specifically, after receiving the instruction to obtain the mask parameters of the caption group sent by the caption decoding module, the video frame gamut interpretation module may send an instruction to obtain the video frame corresponding to the caption group to the video decoding module, and the instruction is used to instruct the video decoding module to send the video frame corresponding to the caption group to the video frame gamut interpretation module. The instruction may carry time information of the caption frame, and the time information of the caption frame may be used by the video decoding module to find the video frame corresponding to the caption group.

S311 and S312: The video decoding module on the electronic device 100 searches for a video frame corresponding to the caption group and sends the video frame corresponding to the caption group to the video frame gamut interpretation module on the electronic device 100.

Specifically, after the video decoding module receives an instruction to obtain a video frame corresponding to a caption group sent by the video frame gamut interpretation module, the video decoding module may find a video frame corresponding to the caption group based on the time information of the caption frame carried in the instruction. Since the video decoding module has obtained the time information of all video frames through decoding in the video decoding phase, the video decoding module can match the time information of all video frames with the time information of the caption frame. If the matching is successful (i.e., if the time information of the video frame matches the time information of the caption frame), the video frame is a video frame corresponding to the caption group. Then, the video decoding module may send the video frame corresponding to the caption group to the video frame gamut interpretation module.

S313: The video frame gamut interpretation module on the electronic device 100 obtains gamut information at the position of each caption in the video frame corresponding to the caption group based on the caption position information in the caption group information.

Specifically, after obtaining a video frame corresponding to a caption group, the video frame gamut interpretation module may determine a video frame region corresponding to the position of each caption based on each caption position information in the caption group information. Furthermore, the video frame gamut interpretation module may calculate gamut information of the video frame region corresponding to the position of each caption.

The following describes in detail the process by which the video frame gamut interpretation module calculates the gamut information of the video frame region corresponding to each caption position.

Assume that the caption "W S Y T K L D G S Y D Z M" in the picture shown in FIG. 2C is caption 1. An example in which the video frame gamut interpretation module calculates the gamut information of the video frame region corresponding to caption 1 is used for explanation.

As shown in FIG. 5, the video frame region corresponding to the position of caption 1 may be the inner region of the rectangular solid box at the top of FIG. 5. Because pixel regions of different color gamuts may exist within one video frame region, one video frame region may be divided into multiple sub-regions, and each sub-region may be referred to as a video frame gamut extraction unit. The division of the sub-regions may be performed based on a preset width, or may be divided based on the width of each character in the caption. For example, caption 1 has a total of 13 characters. In this case, in FIG. 5, the video frame region corresponding to the position of caption 1 is divided into 13 sub-regions, i.e., 13 video frame gamut extraction units, based on the width of each character in caption 1.

Furthermore, the video frame gamut interpretation module may calculate the gamut information of all sub-regions sequentially from left to right (or right to left). Using the calculation of the gamut information of one sub-region in the video frame region as an example, the video frame gamut interpretation module may obtain the color values of all pixels in the sub-region, and then perform overlapping and averaging on the color values of all pixels to obtain an average value of the color values of all pixels in the sub-region. The average value is the color value of the sub-region, and the color value of the sub-region is the gamut information of the sub-region.

は、以下である：

For example, assuming a subregion is m pixels wide and n pixels high, the subregion has a total of m*n pixels, and the color value x of each pixel can be expressed as (r,g,b), where the average color value of all pixels in the subregion is

is the following:

は、ｉ番目のピクセルの赤色値であり、

は、ｉ番目のピクセルの緑色値であり、

は、ｉ番目のピクセルの青色値である。 r _i is the average red value of all pixels in the subregion, g _i is the average green value of all pixels in the subregion, and b _i is the average blue value of all pixels in the subregion.

is the red value of the i-th pixel,

is the green color value of the i-th pixel,

is the blue value of the i-th pixel.

Similarly, the video frame gamut interpretation module may calculate gamut information for all sub-regions of the video frame region corresponding to each caption position, i.e., gamut information for the caption positions in the video frame corresponding to the caption group.

It should be understood that the number of sub-regions into which the video frame region corresponding to the caption is divided may be determined according to a preset division rule, which is not limited in this embodiment of the present application.

Optionally, the color gamut information for a video frame region may also include information such as the luminance of the video frame region.

It should be noted that the above process of calculating color gamut information for the video frame region corresponding to each caption position is only a possible implementation and other implementations may be used. This is not a limitation in this embodiment of the present application.

S314: The video frame gamut interpretation module on the electronic device 100 generates a superimposed caption recognition analysis result based on each caption gamut information in the caption group information and the gamut information at each caption position in the video frame corresponding to the caption group.

Specifically, after calculating the color gamut information at the caption position in the video frame corresponding to the caption group, the video frame color gamut interpretation module may perform an overlay caption recognizability analysis based on the caption color gamut information in the caption group information and the color gamut information at the caption position in the video frame corresponding to the caption group. Furthermore, an overlay caption recognizability analysis result may be generated through the overlay caption recognizability analysis, and the result is used to indicate the magnitude of recognizability (also referred to as the discrimination magnitude) of each caption in the caption group.

In other words, the video frame color gamut interpretation module may determine the difference between the color of the caption and the color of the video frame region corresponding to the caption after the caption group is superimposed on the caption position in the video frame corresponding to the caption group. If the difference is small, it indicates that the caption has low recognizability and is not easily recognized by the user.

The following describes in detail the process by which the video frame gamut interpretation module performs overlay caption recognition analysis.

The video frame color gamut interpretation module may determine a color difference value between the color of the caption and the color of the video frame region corresponding to the caption, the color difference value being used to indicate the difference between the color of the caption and the color of the video frame region corresponding to the caption. The color difference value may be determined by using a related algorithm in the prior art.

In a possible implementation, the color difference value Diff may be calculated using the following formula:

k is the number of all sub-regions of the video frame region that correspond to the caption, _ri is the average red value of all pixels in the sub-region, g _i is the average green value of all pixels in the sub-region, _bi is the average blue value of all pixels in the sub-region, r ₀ is the red value of the caption, g ₀ is the green value of the caption, and b ₀ is the blue value of the caption.

Furthermore, after obtaining the color difference value through calculation, the video frame color gamut interpretation module may determine the magnitude of the caption recognizability by determining whether the color difference value is less than a pre-set color difference threshold.

If the color difference value is less than a pre-set color difference threshold (sometimes called a first threshold), it indicates low caption recognition.

In some embodiments, the degree of caption recognition may be further determined with reference to the luminance of the area of the video frame that corresponds to the caption.

For example, although the color difference value between the caption color of the caption "Caption with unclear color" shown in FIG. 2C and the color of the video frame area corresponding to the caption is not very small, the problem of low caption recognizability still exists because the luminance of the video frame area corresponding to the caption is too high. Therefore, in this case, the caption recognizability can be further determined with reference to the luminance of the video frame area corresponding to the caption. If the luminance of the video frame area corresponding to the caption is higher than the preset luminance threshold, it indicates low caption recognizability.

For a solid color caption, the extracted caption color gamut information may include only one parameter, i.e., one color value corresponding to the caption. For a non-solid color caption, the extracted caption color gamut information may include multiple parameters. For example, for a gradient color caption, the extracted caption color gamut information may include multiple parameters such as a start point color value, an end point color value, and a gradient direction. In this case, in a possible implementation, an average value of the start point color value and the end point color value of the caption may be calculated first, and then the average value is used as the color value corresponding to the caption to perform the overlay caption recognition analysis.

It should be noted that the above process in which the video frame gamut interpretation module performs the overlay caption recognition analysis is only a possible implementation and other implementations may be used. This is not a limitation in this embodiment of the present application.

S315: The video frame color gamut interpretation module on the electronic device 100 calculates the color value and transparency of a mask corresponding to each caption in the caption group based on the overlay caption recognition analysis result.

Specifically, after generating the overlay caption recognition analysis result, the video frame color gamut interpretation module may calculate the color value and transparency of a mask corresponding to each caption in the caption frame based on the result.

For a highly recognizable caption (e.g., the caption "Highly Recognizable Caption" or the caption "Caption Synchronized with Audio" in FIG. 2C), the color value of the mask corresponding to that caption may be a preset fixed value and the transparency may be set to 100%.

For captions with low recognition (e.g., the caption "W S Y T K L D G S Y D Z M" or the caption "Obscure color caption" in FIG. 2C), the color value and transparency of the mask corresponding to the caption needs to be further determined based on the color gamut information of the caption or the color gamut information of the video frame region corresponding to the position of the caption.

There may be many specific methods for determining the color value and transparency of the mask corresponding to the caption. This is not limited in this embodiment of the present application, and those skilled in the art may select the method according to the requirements.

In a possible implementation, a color value corresponding to a color having the largest color difference value from the color value of the caption or the color value of the video frame region corresponding to the caption may be determined as the color value of the mask corresponding to the caption. In this way, the user can see the caption more clearly. Alternatively, a color value corresponding to a color having an intermediate color difference value from the color value of the caption or the color value of the video frame region corresponding to the caption may be determined as the color value of the mask corresponding to the caption, thereby ensuring that the user can see the caption clearly while avoiding eye discomfort caused by large color differences, etc.

For example, the electronic device 100 may calculate a color difference value Diff between the color value corresponding to each color in the color value table and the color value of the caption, and then select the color value corresponding to the color with the maximum/middle color difference value Diff as the color value of the mask. In a possible implementation, the color difference value Diff between the color value corresponding to each color in the color value table and the color value of the caption may be calculated by using the following formula:

Assuming the color value corresponding to a particular color in the color value table is ( _R0 , _G0 , _B0 ), _R0 is the red value corresponding to that color, _G0 is the green value corresponding to that color, _B0 is the blue value corresponding to that color, _r0 is the red value of the caption, _g0 is the green value of the caption, and _b0 is the blue value of the caption.

As another example, the electronic device 100 may calculate a color difference value Diff between the color value corresponding to each color in the color value table and the color value of the video frame region corresponding to the caption, and then select the color value corresponding to the color with the maximum/middle color difference value Diff as the color value of the mask. In a possible implementation, the color difference value Diff between the color value corresponding to each color in the color value table and the color value of the video frame region corresponding to the caption may be calculated by using the following formula:

Assuming the color value corresponding to a particular color in the color value table is ( _R0 , _G0 , _B0 ), _R0 is the red value corresponding to that color, _G0 is the green value corresponding to that color, and _B0 is the blue value corresponding to that color. k is the number of all sub-regions of the video frame region corresponding to the caption, _ri is the average red value of all pixels in the sub-region, and g _i is the average green value of all pixels in the sub-region.

In a possible implementation, the transparency of the mask corresponding to the caption may be further determined based on the color value of the mask corresponding to the caption. For example, if the color value of the mask corresponding to the caption is significantly different from the color value of the caption, a large value (e.g., a value larger than 50%) may be selected for the transparency of the mask corresponding to the caption, ensuring that the user can clearly see the caption while reducing the occlusion of the video picture by the caption overlapping region.

S316: The video frame color gamut interpretation module on the electronic device 100 sends the color values and transparency of the masks corresponding to each caption in the caption group to the caption decoding module on the electronic device 100.

Specifically, after calculating the color values and transparency of the mask corresponding to each caption in the caption group, the video frame color gamut interpretation module may send the color values and transparency of the mask corresponding to each caption in the caption group to the caption decoding module, and the caption position information of the caption corresponding to the mask may also be conveyed, so that the caption decoding module may map the caption to the mask one-to-one.

S317: The caption decoding module on the electronic device 100 generates a corresponding mask based on the color value and transparency of the mask corresponding to each caption in the caption group, and superimposes each caption in the caption group with the corresponding mask to generate a caption frame with a mask.

Specifically, after receiving the color value and transparency of the mask corresponding to each caption in the caption group sent by the video frame color gamut interpretation module, the caption decoding module may generate a mask corresponding to the caption (e.g., a mask corresponding to caption 1 shown in FIG. 5) based on the color value and transparency of the mask corresponding to the caption and the caption position information of the caption. The shape of the mask may be rectangular or any other shape that can cover the caption. This is not limited in this embodiment of the present application.

Similarly, for each caption in the caption group, the caption decoding module may generate a mask corresponding to the caption.

For example, as shown in FIG. 2C, it is easy to see that there are four captions in the picture. Therefore, the caption decoding module may generate four masks, one caption corresponding to one mask.

Furthermore, the caption decoding module may overlay the caption on top of a mask corresponding to the caption to generate a caption with a mask (e.g., Caption with Mask 1 shown in FIG. 5).

Similarly, the caption decoding module may overlay each caption in a caption group onto a corresponding mask to generate a caption frame with a mask.

Figure 6A shows an example of a caption frame with a mask. It can be seen that each caption has a mask superimposed on it. The masks corresponding to the captions with high recognizability (e.g. "High Recognizability Caption" or "Synchronized with Audio Caption") have 100% transparency, whereas the masks corresponding to the captions with low recognizability (e.g. "W S Y T K L D G S Y D Z M" or "Obscure Color Caption") have less than 100% transparency and certain color values are present.

S318: The caption decoding module on the electronic device 100 sends the caption frame with the mask to the video frame synthesis module on the electronic device 100.

Specifically, after generating the caption frame with the mask, the caption decoding module may send the caption frame with the mask to a video frame synthesis module, which then generates the video frame to be displayed.

Phase 4: Compositing, rendering, and displaying video frames

S319 and S320: The video frame synthesis module on the electronic device 100 overlays and combines the received video frame with the caption frame having the mask to generate the video frame to be displayed, and sends the video frame to be displayed to a video frame queue on the electronic device 100.

S321 to S323: The video rendering module may read the video frames to be displayed from the video frame queue in chronological order and render the video frames to be displayed in chronological order to generate rendered video frames.

S324: The electronic device 100 displays the rendered video frame.

For the specific execution process of steps S319 to S324, please refer to the relevant contents of steps S109 to S114 in the embodiment shown in Figures 1A and 1B. Details will not be described again here.

It should be noted that in some embodiments, the video decoding module, the caption decoding module, the video frame gamut interpretation module, the video frame synthesis module, the video frame queue, and the video rendering module may alternatively be integrated into a video application to perform the caption display method provided in this embodiment of the present application. This is not limited to this embodiment of the present application.

For example, FIG. 6B may be a picture of a frame in a rendered video frame displayed after the electronic device 100 executes the caption display method shown in FIG. 3A, FIG. 3B, FIG. 3C, and FIG. 3D (one caption may correspond to one mask). Compared with the picture shown in FIG. 2C, it is easy to see that the recognition degree of the caption "W S Y T K L D G S Y D Z M" and the recognition degree of the caption "obscure color caption" are greatly improved after the corresponding mask is added to the caption group. In addition, the mask corresponding to the caption has a certain transparency, so that the caption overlapping area does not completely occlude the video picture. In this way, the effects of video display and caption display are comprehensively considered, thereby ensuring that the user can clearly see the caption while ensuring a certain visibility of the video picture without changing the color of the caption selected by the user, thereby improving the user experience.

In addition, the position of the caption, the color of the video background, etc. may change during the entire video playback process. Therefore, the above-mentioned caption display method may be always performed so that the user can clearly see the caption during the entire video playback process. For example, FIG. 6B may be a schematic diagram of a first user interface where the video playback progress is at the moment 8:00, and FIG. 6C may be a schematic diagram of a second user interface where the video playback progress is at the moment 8:02. The video frames included in the first user interface are different from the video frames included in the second user interface. As shown in FIG. 6C, compared with FIG. 6B, it can be seen that the caption "W S Y T K L D G S Y D Z M", the caption "Highly recognizable caption", and the caption "Obscure color caption" are all moved to the left side of the display screen. The electronic device 100 recalculates the color value and transparency of the mask corresponding to the caption based on the color value of the caption and the color value of the current video frame area corresponding to the caption to generate a mask corresponding to the caption. It can be easily seen that in the second user interface, the video background color of the caption "W S Y T K L D G S Y D Z M" corresponding to the current video frame area is changed, and the recognition degree of the caption is also increased. Therefore, compared with FIG. 6B, the mask corresponding to the caption "W S Y T K L D G S Y D Z M" is also changed. It can be seen that no mask is displayed for the caption. Specifically, the transparency of the mask corresponding to the caption is changed to 100%, or there is no mask for the caption.

The video playback pictures shown in Figures 6B and 6C may be displayed in full screen mode or in partial screen mode. This is not a limitation in this embodiment of the present application.

The mask corresponding to the caption shown in FIG. 6B is a mask that spans the region where the entire caption is located, i.e., one caption corresponds to only one mask. In some practical application scenarios, one caption may span multiple regions with large color gamut differences. As a result, some parts of the caption are highly recognizable and other parts of the caption are less recognizable. In this case, multiple corresponding masks may be generated for one caption. For example, in the caption "W S Y T K L D G S Y D Z M" shown in FIG. 2C, the caption recognition is low at the beginning of the caption region (i.e., the four characters "W S Y T" are difficult for users to recognize), and the caption recognition is also low at the end of the caption region (i.e., the four characters "Y D Z M" are difficult for users to recognize). The middle part of the caption region has high caption recognition (i.e., the four letters "K L D G S" are easy to recognize by users). Therefore, in this case, one corresponding mask may be generated for each of the beginning, middle, and end parts of the caption region, i.e., the caption may have three corresponding masks.

For the aforementioned application scenario where one caption corresponds to multiple masks, in this embodiment of the present application, some corresponding improvements may be made to steps S313 to S317 based on the methods shown in Figures 3A, 3B, 3C, and 3D so that one caption corresponds to multiple masks. Other steps do not need to be changed.

Below we explain in detail the process by which one caption corresponds to multiple masks.

In the process of generating color gamut information at the caption position in the video frame corresponding to the caption group, the video frame gamut interpretation module may sequentially calculate the color values of all sub-regions in order from left to right (or from right to left). In the aforementioned application scenario where one caption needs to correspond to multiple masks, i.e., one caption spans multiple regions with large gamut differences, the video frame gamut interpretation module may compare the color values of adjacent sub-regions, and if the color values of the adjacent sub-regions are close, the adjacent sub-regions are combined into one region, and the combined region corresponds to one mask. If the color values of the adjacent sub-regions are significantly different, the adjacent sub-regions are not combined, and the two uncombined regions correspond to respective masks. Thus, one caption may correspond to multiple masks.

As shown in FIG. 7A, when one caption can correspond to multiple masks, steps S313 to S317 can be specifically executed based on the following steps. In the following explanation, the caption 1 shown in FIG. 7B is the caption "W S Y T K L D G S Y D Z M" shown in FIG. 2C.

S701: The video frame color gamut interpretation module sequentially calculates the color values of all sub-regions of the video frame region corresponding to the position of the caption, and combines the sub-regions with similar color values to obtain M second sub-regions.

Specifically, after calculating the color values of all sub-regions from left to right (or from right to left) sequentially according to step S313, the video frame gamut interpretation module further needs to compare the color values of adjacent sub-regions and combine the sub-regions with similar color values to obtain M second sub-regions, where M is a positive integer. As shown in Fig. 7B, after comparing the color values of adjacent sub-regions and combining the sub-regions with similar color values, the video frame gamut interpretation module divides the video frame region corresponding to the position of the caption into three regions (i.e., three second sub-regions), namely, region A, region B, and region C. Assume that region A is formed by combining a sub-regions, region B is formed by combining b sub-regions, and region C is formed by combining c sub-regions.

Color values being close may mean that the difference value between the color values of the two subregions is less than a second threshold, which is preset.

S702: The video frame gamut interpretation module performs overlay caption recognizability analysis separately for the M second sub-regions to generate overlay caption recognizability analysis results for the M second sub-regions.

Specifically, instead of directly performing the overlay caption recognizability analysis on the entire video frame region, the video frame gamut interpretation module needs to perform the overlay caption recognizability analysis separately on region A, region B, and region C. Similarly, the video frame gamut interpretation module may use the color difference values in step S314 to perform the overlay caption recognizability analysis separately on region A, region B, and region C. The process is as follows:

Color difference value Diff1 of area A:

a is the number of subregions contained in region A, r _i is the average red value of all pixels in the subregions in region A, g _i is the average green value of all pixels in the subregions in region A, and b _i is the average blue value of all pixels in the subregions in region A. r ₀ is the red value of the caption in region A, g ₀ is the green value of the caption in region A, and b ₀ is the blue value of the caption in region A.

Color difference value Diff2 of area B:

b is the number of subregions contained in region B, r _i is the average red value of all pixels in the subregions in region B, g _i is the average green value of all pixels in the subregions in region B, and b _i is the average blue value of all pixels in the subregions in region B. r ₀ is the red value of the caption in region B, g ₀ is the green value of the caption in region B, and b ₀ is the blue value of the caption in region B.

Color difference value Diff3 of area C:

c is the number of subregions contained in region C, r _i is the average red value of all pixels in the subregions in region C, g _i is the average green value of all pixels in the subregions in region C, and b _i is the average blue value of all pixels in the subregions in region C. r ₀ is the red value of the caption in region C, g ₀ is the green value of the caption in region C, and b ₀ is the blue value of the caption in region C.

After separately obtaining the color difference values of area A, area B, and area C through calculation, the video frame color gamut interpretation module may determine whether the color difference values of the three areas are less than the preset color difference threshold. If the color difference value of the area is less than the preset color difference threshold, it indicates that the caption recognition degree of the area is low.

S703: The video frame color gamut interpretation module determines color values and transparency of masks corresponding to the M second sub-regions, respectively, based on the superimposed caption recognition analysis results of the M second sub-regions and the caption color gamut information.

Specifically, the video frame color gamut interpretation module needs to determine the color value and transparency of the mask corresponding to region A, the color value and transparency of the mask corresponding to region B, and the color value and transparency of the mask corresponding to region C based on the caption color gamut information of region A, region B, and region C and the result of the superimposed caption recognition analysis. The specific process of determining the color value and transparency of the mask corresponding to each second sub-region is similar to the process of determining the color value and transparency of the mask corresponding to the entire video frame region corresponding to the position of the caption in step S315. Please refer to the related contents mentioned above. Details will not be described again here.

S704: The video frame gamut interpretation module sends the color values, transparency, and position information of the masks corresponding to the M second sub-regions to the caption decoding module.

Specifically, since one caption may correspond to multiple masks, the video frame gamut interpretation module not only needs to send the color value and transparency of the mask corresponding to each caption in the caption group to the caption decoding module, but also needs to send the position information of each mask (or the position information of each mask relative to the corresponding caption) to the caption decoding module. The position information of each mask can be obtained based on the caption position information. Specifically, when one caption corresponds to multiple masks, since the caption position information is known, the position information of all sub-regions in the video frame region at the caption position can be estimated. Furthermore, the position information of the mask corresponding to each second sub-region can be estimated.

S705: The caption decoding module generates a mask corresponding to the caption based on the color values, transparency, and position information of the mask corresponding to the M second sub-regions, and superimposes the caption on the mask to generate a caption with the mask.

Specifically, in the case of one caption corresponding to multiple masks, the caption decoding module may generate three masks corresponding to the caption (e.g., a mask corresponding to caption 1 shown in FIG. 7B) based on the color value, transparency, and mask position information of the mask of each second sub-region corresponding to the caption. Then, the caption decoding module may overlay the caption on a layer above the mask corresponding to the caption to generate a caption with a mask (e.g., caption 1 with mask shown in FIG. 7B).

As shown in FIG. 2C, the three captions “highly recognizable caption”, “obscure color caption”, and “synchronized with audio caption” do not span multiple regions with large color gamut differences, so each of the three captions still corresponds to one mask.

The caption decoding module may overlay each caption in the caption group onto a corresponding mask to generate a caption frame with the mask.

8A shows an example of a caption frame with masks. It can be seen that three masks are superimposed on the caption "W S Y T K L D G S Y D Z M". The recognizability of "W S Y T" and the recognizability of "Y D Z M" are low, so the transparency of the corresponding mask is less than 100% and there is a specific color value. The recognizability of "K L D G S" is high, so the transparency of the corresponding mask is 100%. One mask is superimposed on each of the other three captions. The recognizability of the caption "High recognizability caption" and the recognizability of the caption "Caption synchronized with audio" are high, so the transparency of the corresponding mask is 100%. The recognizability of the caption "Caption with unclear color" is low, so the transparency of the corresponding mask is less than 100% and there is a specific color value.

For example, Fig. 8B may be a picture of a frame in the rendered video frame displayed after the electronic device 100 executes the improved caption display method shown in Figs. 3A, 3B, 3C, and 3D (a caption spanning multiple regions with large color gamut differences may correspond to multiple masks). Compared to the picture shown in Fig. 6B, the caption "W S Y T K L D G S Y D Z M" spans multiple regions with large color gamut differences, so the mask corresponding to the caption has changed. It can be easily seen that the middle part of the caption region (i.e., the portion "K L D G S") has a high degree of caption recognition, so the transparency of the mask corresponding to that part may be set to 100% (i.e., completely transparent) or no mask may be set. Since the leading portion (i.e., portion "W S Y T") and trailing portion (i.e., portion "Y D Z M") of the caption region have low caption recognition degree, the color value and transparency of the mask corresponding to these two portions are calculated based on the caption color gamut information and the color gamut information of the region where the two portions are located, respectively. In this way, the transparency of the mask corresponding to the middle portion of the caption "W S Y T K L D G S Y D Z M" region can be 100% or no mask can be set, so that the mask further reduces the occlusion of the video picture, and the user experience is further improved, based on which the beneficial effect shown in FIG. 6B is achieved.

In addition, the position of the caption, the color of the video background, etc. may change during the entire video playback process. Therefore, the above-mentioned caption display method may be always performed so that the user can clearly see the caption during the entire video playback process. For example, FIG. 8B may be a schematic diagram of a user interface where the video playback progress is at the moment 8:00 and a first video frame is included. FIG. 8C may be a schematic diagram of a user interface where the video playback progress is at the moment 8:01 and a second video frame is included. The first video frame is the same as the second video frame. As shown in FIG. 8C, compared with FIG. 8B, it can be seen that the caption "W S Y T K L D G S Y D Z M", the caption "Highly recognizable caption", and the caption "Obscure color caption" have all moved to the left side of the display screen. The electronic device 100 recalculates the color value and transparency of the mask corresponding to the caption based on the color value of the caption and the color value of the current video frame area corresponding to the caption to generate a mask corresponding to the caption. Compared with FIG. 8B, it is easy to see that the mask corresponding to the caption "W S Y T K L D G S Y D Z M" in FIG. 8C has changed significantly. In FIG. 8B, the parts with low caption recognition are "W S Y T" and "Y D Z M". Therefore, the masks corresponding to these two parts have certain color values, and the transparency of the corresponding masks is less than 100%. The part with high caption recognition is "K L D G S", so no mask is displayed in this part. Specifically, the transparency of the mask corresponding to the caption may be set to 100%, or no mask may be set. However, in FIG. 8C, the part with low caption recognition has changed to "W S Y T K" and "D Z M". Therefore, the electronic device 100 recalculates the color values and transparency of the masks corresponding to these two parts based on the color values of the caption and the color values of the current video frame area corresponding to the caption. The recognition of the two parts is low, so the masks corresponding to the two parts have certain color values. In addition, the transparency of the corresponding masks is less than 100%. The part with high caption recognition has changed to "L D G S Y". Therefore, no mask is displayed in this part. Specifically, the transparency of the mask corresponding to the caption may be set to 100%, or no mask may be set. The process of generating a mask corresponding to the caption in FIG. 8C is similar to the process of generating a mask corresponding to the caption in FIG. 8B. Details will not be described again here.

The video playback pictures shown in Figures 8B and 8C may be displayed in full screen mode or in partial screen mode. This is not a limitation in this embodiment of the present application.

In this embodiment of the application, for a highly recognizable caption, the electronic device 100 generates a mask for the caption, where the color value of the mask may be a preset color value and the transparency of the mask is 100%. In some embodiments, for a highly recognizable caption, the electronic device 100 may alternatively not generate a mask for the caption. Specifically, if the electronic device 100 determines that the caption has a high degree of recognition, the electronic device 100 may not perform further processing on the caption, and thus the caption does not have a corresponding mask, i.e., no mask is set for the caption.

In this embodiment of the application, a caption corresponds to a mask (i.e., a caption corresponds to a group of mask parameters) may mean that a caption corresponds to a mask including a color value and transparency. A caption corresponds to multiple masks (i.e., a caption corresponds to multiple groups of mask parameters) may mean that a caption corresponds to multiple masks with different color values and different transparency, and that a caption corresponds to a mask with different color values and different transparency (i.e., multiple masks with different color values and different transparency are combined into a mask with different color values and different transparency).

In this embodiment of the present application, a mobile phone is used as an example of the electronic device 100. Alternatively, the electronic device 100 may be a portable electronic device such as a tablet computer (Pad), a personal digital assistant (PDA), or a laptop computer (Laptop). The type, physical form, and size of the electronic device 100 are not limited in this embodiment of the present application.

In an embodiment of the present application, the first video may be a video played by the electronic device 100 after the user taps the video playback option 221 shown in FIG. 2B. The first interface may be the user interface shown in FIG. 6B. The first picture may be a picture of the video frame shown in FIG. 6B. The first caption may be the caption "W S Y T K L D G S Y D Z M". The first region is a region in the first picture that corresponds to the display position of the first caption. The first value may be a color difference value between the color of the first caption and the color of the first picture region that corresponds to the display position of the first caption. The second interface may be the user interface shown in FIG. 6C. The second picture may be a picture of the video frame shown in FIG. 6C. The second region is a region in the second picture that corresponds to the display position of the first caption. The second value may be a color difference value between the color of the first caption and the color of the second picture region corresponding to the display position of the first caption. The first video file may be a video file corresponding to the first video, and the first caption file may be a caption file corresponding to the first video. The first video frame is a video frame used to generate the first picture. The first caption frame is a caption frame that includes the first caption and carries the same time information as the first video frame, and the second caption frame is a caption frame generated after the first caption is superimposed on the first mask (i.e., a caption frame with a mask). The first sub-region may be a video frame gamut extraction unit. The second sub-region may be a region (e.g., region A, region B, and region C) obtained after adjacent first sub-regions with close color values are combined. The first sub-mask may be a mask corresponding to each second sub-region. The first mask may be a mask corresponding to the caption "W S Y T K L D G S Y D Z M" shown in Figure 6B or may be a mask corresponding to the caption "W S Y T K L D G S Y D Z M" shown in Figure 8B. The third interface may be the user interface shown in Figure 8B and the third picture may be a picture of the video frame shown in Figure 8B. The first portion may be "W S Y T" in the caption "W S Y T K L D G S Y D Z M" and the second portion may be "K L D G S" in the caption "W S Y T K L D G S Y D Z M". The second submask may be a mask corresponding to "W S Y T" (i.e., the mask of area A shown in FIG. 7B). The third submask may be a mask corresponding to "K L D G S" (i.e., the mask of area B shown in FIG. 7B). The second mask may be a mask corresponding to the caption "W S Y T K L D G S Y D Z M" shown in FIG. 6C.

The structure of the electronic device 100 according to one embodiment of the present application is described below.

Figure 9 shows an example of the configuration of an electronic device 100 according to one embodiment of the present application.

As shown in FIG. 9, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a loudspeaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identity module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, an optical proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

It may be understood that the structure shown in this embodiment of the present application does not constitute a specific limitation on the electronic device 100. In some other embodiments of the present application, the electronic device 100 may include more or fewer components than those shown in the figures, some components may be combined, some components may be divided, or different component arrangements may be used. The components shown in the figures may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and/or a neural-network processing unit (NPU), etc. The different processing units may be separate devices or may be integrated into one or more processors.

The controller may be the nerve center and command center of the electronic device 100. The controller may generate operation control signals based on instruction operation codes and timing signals to exercise control over instruction fetching and execution.

Memory may further be disposed in the processor 110 and configured to store instructions and data. In some embodiments, the memory in the processor 110 is a cache. The memory may store instructions or data that have just been used or that are used periodically by the processor 110. When the processor 110 needs to use the instructions or data again, the processor 110 may retrieve the instructions or data directly from the memory. This avoids repeated accesses, reducing the latency of the processor 110 and improving system efficiency.

In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input/output (GPIO) interface, a subscriber identity module (SIM) interface, a universal serial bus (USB) port, and/or the like.

The ^I2C interface is a bidirectional synchronous serial bus including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple ^I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the camera flash, the camera 193, etc., via different ^I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K via an I2C interface, and the processor 110 and the touch sensor 180K communicate with each other via the I2C bus interfaces, thereby implementing the touch function of the electronic device 100.

The ^I2S interface may be used for audio communication. In some embodiments, the processor 110 may include multiple ^I2S buses. The processor 110 may be coupled to the audio module 170 via the ^I2S bus to facilitate communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 may transmit audio signals to the wireless communication module 160 via the ^I2S interface to implement a function for answering a call via a Bluetooth headset.

The PCM interface may also be used for audio communication and may perform sampling, quantization, and encoding on the analog signal. In some embodiments, audio module 170 and wireless communication module 160 may be coupled via a PCM interface . In some embodiments, audio module 170 may also send audio signals to wireless communication module 160 via the PCM interface to implement the functionality of answering a call via a Bluetooth headset. Both the I2S interface and the PCM interface may be used for audio communication.

The UART interface is a universal serial data bus and is used for asynchronous communication. The bus may be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is configured to connect generally to the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with a Bluetooth module in the wireless communication module 160 through the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 may send audio signals to the wireless communication module 160 through the UART interface to implement the functionality of playing music through a Bluetooth headset.

The MIPI interface may be configured to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), and the like. In some embodiments, the processor 110 communicates with the camera 193 via the CSI interface to implement the image capture function of the electronic device 100. The processor 110 communicates with the display 194 via the DSI interface to implement the display function of the electronic device 100.

The GPIO interface may be configured by software. The GPIO interface may be configured as a control signal or a data signal. In some embodiments, the GPIO interface may be configured to connect the processor 110 to the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface may be further configured as an ^I2C interface, an ^I2S interface, a UART interface, a MIPI interface, etc.

The USB interface 130 is an interface that complies with the USB standard specifications, and specifically may be a mini USB interface, a micro USB interface, a USB Type-C interface, etc. The USB interface 130 may be configured to connect to a charger to charge the electronic device 100, and may also be configured to transmit data between the electronic device 100 and a peripheral device. It may also be configured to connect to a headset to play audio through the headset. The interface may also be configured to connect to another terminal device, such as an AR device.

It may be understood that the interface connection relationships between modules shown in this embodiment of the present application are merely exemplary illustrations and do not constitute limitations on the structure of the electronic device 100. In some other embodiments of the present application, the electronic device 100 may alternatively use a different interface connection scheme than the aforementioned embodiment or may use a combination of multiple interface connection schemes.

The charging management module 140 is configured to receive a charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive a charging input from a wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive a wireless charging input via a wireless charging coil of the electronic device 100. The charging management module 140 may further provide power to the electronic device 100 via the power management module 141 while charging the battery 142.

The power management module 141 is configured to connect to the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives inputs from the battery 142 and/or the charging management module 140 and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, the wireless communication module 160, and the like. The power management module 141 may be further configured to monitor parameters such as battery capacity, battery cycle amount, and battery health status (leakage and impedance). In some other embodiments, the power management module 141 may also be provided within the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 may be provided within the same device.

The wireless communication function of the electronic device 100 may be embodied via antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, a modem processor, a baseband processor, etc.

Antenna 1 and Antenna 2 are configured to transmit and receive electromagnetic signals. Each antenna in electronic device 100 may be configured to cover one or more communication frequency bands. Different antennas may be further multiplexed to improve antenna utilization. For example, Antenna 1 may be multiplexed as a diversity antenna in a wireless local area network. In some other embodiments, the antennas may be used in conjunction with a tuning switch.

The mobile communication module 150 may provide a solution for wireless communication including 2G, 3G, 4G, 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 may receive electromagnetic waves via the antenna 1, perform processing such as filtering and amplification on the received electromagnetic waves, and send the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 150 may further amplify the signal modulated by the modem processor and convert the signal to electromagnetic waves by using the antenna 1 for radiation. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be located in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be located in the same device as at least some of the modules of the processor 110.

The modem processor may include a modulator and a demodulator. The modulator is configured to modulate a low-frequency baseband signal to be transmitted into an intermediate frequency signal and a high-frequency signal. The demodulator is configured to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs an audio signal via an audio device (such as, but not limited to, loudspeaker 170A, receiver 170B, etc.) or displays an image or video via display 194. In some embodiments, the modem processor may be a separate device. In some other embodiments, the modem processor may be independent of the processor 110 and may be provided in the same device as the mobile communication module 150 or another functional module.

The wireless communication module 160 may be applied to the electronic device 100 to provide wireless communication solutions including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC) technology, infrared (IR) technology, etc. The wireless communication module 160 may be one or more components integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signal, and transmits the processed signal to the processor 110. The wireless communication module 160 may further receive signals to be transmitted from the processor 110, perform frequency modulation and amplification on the signals, and convert the signals into electromagnetic waves for emission via the antenna 2.

In some embodiments, antenna 1 and mobile communication module 150 are coupled in electronic device 100, and antenna 2 and wireless communication module 160 are coupled in electronic device 100, such that electronic device 100 can communicate with a network and another device by using wireless communication technologies. Wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), Bluetooth ( BT ) , GNSS, WLAN, NFC, FM, IR technologies, and/or the like. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a BeiDou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS), and/or a satellite based augmentation systems (SBAS).

The electronic device 100 implements display functionality by using a GPU, a display 194, an application processor, and the like. The GPU is a microprocessor for image processing and is connected to the display 194 and the application processor. The GPU is configured to perform mathematical and geometric calculations and render images. The processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

The display 194 is configured to display images, videos, and the like. The display 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini-LED, a micro-LED, a micro-OLED, a quantum dot light-emitting diodes (QLED), and the like. In some embodiments, the electronic device 100 can include one or N displays 194, where N is a positive integer greater than 1.

The electronic device 100 may implement the imaging function through an ISP, a camera 193, a video codec, a GPU, a display 194, an application processor, etc.

The ISP is configured to process data fed back by the camera 193. For example, when taking a picture, a shutter is opened and a light beam is sent through a lens to the camera's photosensitive elements. The light signal is converted into an electrical signal. The camera's photosensitive elements send the electrical signal to the ISP for processing, which converts the electrical signal into a visible image. The ISP may further perform algorithmic optimization for image noise, brightness, and skin tone. The ISP may further optimize parameters such as exposure and color temperature in the picture taking scenario. In some embodiments, the ISP may be located within the camera 193.

The camera 193 is configured to capture still images or video. An optical image of an object is generated through a lens and projected onto a photosensitive element. The photosensitive element may be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then sent to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a format such as standard RGB or YUV. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

The digital signal processor is configured to process digital signals, and may process other digital signals in addition to the digital image signal. For example, if the electronic device 100 selects a frequency, the digital signal processor is configured to perform a Fourier transform or the like on the frequency energy.

A video codec is configured to compress or decompress digital video. Electronic device 100 may support one or more video codecs. In this manner, electronic device 100 can play or record video in multiple encoding formats, e.g., moving picture experts group (MPEG)-1, MPEG-2, MPEG-3, and MPEG-4.

The NPU is a neural-network (NN) computing processor that can simulate a biological neural network structure, such as the communication mode between neurons in the human brain, to perform high-speed processing on input information and perform continuous self-learning. The NPU can implement applications such as intelligent recognition of the electronic device 100, such as image recognition, face recognition, speech recognition, and text understanding.

The external memory interface 120 may be configured to connect to an external memory card, e.g., a microSD card, to expand the storage capabilities of the electronic device 100. The external memory card communicates with the processor 110 by using the external memory interface 120 to implement data storage functions. The external memory card stores files such as music and videos.

The internal memory 121 may be configured to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store an operating system, applications necessary for at least one function (such as a voice playback function and an image playback function), and the like. The data storage area may store data (such as voice data and a phone book) created based on the use of the electronic device 100, and the like. In addition, the internal memory 121 may include a high-speed random access memory, and may further include a non-volatile memory, such as at least one magnetic storage device, a flash memory device, and a universal flash storage (UFS).

The electronic device 100 may implement audio functions such as music playback and recording via an audio module 170, a loudspeaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, an application processor, etc.

Audio module 170 is configured to convert digital audio information to an analog audio signal output, and is also configured to convert an analog audio input to a digital audio signal. Audio module 170 may be further configured to encode and decode audio signals. In some embodiments, audio module 170 may be located within processor 110, or some functional modules within audio module 170 are located within processor 110.

The loudspeaker 170A, also referred to as a "loudspeaker," is configured to convert audio electrical signals into voice signals. The electronic device 100 may listen to music or answer calls in a hands-free mode through the loudspeaker 170A.

Receiver 170B, also known as an "earpiece," is configured to convert audio electrical signals into speech signals. When answering a call or receiving a voice message via electronic device 100, receiver 170B can be held close to a person's ear to hear the audio.

The microphone 170C is also called a "mike" or "mic" and is configured to convert a voice signal into an electrical signal. When making a phone call or sending a voice message, a user may put a person's mouth close to the microphone 170C and make a sound to input the voice signal into the microphone 170C. At least one microphone 170C may be disposed in the electronic device 100. In some other embodiments, in addition to collecting voice signals, two microphones 170C may be disposed in the electronic device 100 to implement noise reduction functions. In some other embodiments, three, four, or more microphones 170C may alternatively be disposed in the electronic device 100 to collect voice signals, implement noise reduction, identify sound sources, and thereby implement directional recording functions, etc.

The headset jack 170D is configured to connect to a wired headset. The headset jack 170D may be a USB interface 130, a 3.5 mm open mobile terminal platform (OMTP) standard interface, or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.

The pressure sensor 180A is configured to sense a pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A may be disposed within the display 194. There are many types of pressure sensor 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. The capacitive pressure sensor may include at least two parallel plates having a conductive material. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the strength of the pressure based on the change in capacitance. When a touch operation is performed on the display 194, the electronic device 100 detects the strength of the touch operation by using the pressure sensor 180A. The electronic device 100 may also calculate a touch position based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations performed on the same touch position but with different touch operation strengths may correspond to different operation instructions. For example, when a touch operation with a touch operation strength less than a first pressure threshold is performed on a short message application icon, an instruction to view a short message is executed. When a touch operation having a touch operation strength equal to or greater than the first pressure threshold is performed on the short message application icon, a command to create a new short message is executed.

The gyro sensor 180B may be configured to determine the motion attitude of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (x, y, and z) may be determined by using the gyro sensor 180B. The gyro sensor 180B may be used for image stabilization. For example, when a shutter is pressed, the gyro sensor 180B detects the angle at which the electronic device 100 jitters, and based on this angle, calculates the distance that the lens module needs to compensate, allowing the lens to counteract the jitter of the electronic device 100 through a reverse motion to achieve image stabilization. The gyro sensor 180B may further be used in navigation scenarios and motion-sensing gaming scenarios.

The air pressure sensor 180C is configured to measure air pressure. In some embodiments, the electronic device 100 calculates altitude by using the air pressure values measured by the air pressure sensor 180C to aid in positioning and navigation.

The magnetic sensor 180D includes a Hall sensor. The electronic device 100 may detect the opening and closing of the flip holster by using the magnetic sensor 180D. In some embodiments, when the electronic device 100 is a flip device, the electronic device 100 may detect the opening and closing of the flip by using the magnetic sensor 180D. Furthermore, based on the detected opening and closing state of the leather case or the opening and closing state of the flip cover, a function such as automatically unlocking when the flip cover is opened is set.

The acceleration sensor 180E can detect acceleration of the electronic device 100 in various directions (usually three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. The acceleration sensor 180E may be further configured to identify the orientation of the electronic device 100, for applications such as a pedometer and switching between landscape and portrait modes.

The distance sensor 180F is configured to measure distance. The electronic device 100 may measure distance in an infrared or laser manner. In some embodiments, in a photography scenario, the electronic device 100 may use the distance sensor 180F to measure distance and achieve fast focusing.

The optical proximity sensor 180G may include, for example, a light emitting diode (LED) and a photodetector such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light to the outside by using the light emitting diode. The electronic device 100 uses the photodiode to detect infrared reflected light from surrounding objects. If abundant reflected light is detected, it may be determined that an object is present in the vicinity of the electronic device 100. If insufficient reflected light is detected, the electronic device 100 may determine that no object is present in the vicinity of the electronic device 100. The electronic device 100 may use the optical proximity sensor 180G to detect when a user holds the electronic device 100 close to the ear for a call and automatically turn off the display to save power. The optical proximity sensor 180G may also be used in holster mode or pocket mode for automatic unlocking and screen locking.

The ambient light sensor 180L is configured to sense the brightness of the ambient light. The electronic device 100 may adaptively adjust the brightness of the display 194 based on the sensed brightness of the ambient light. The ambient light sensor 180L may further be configured to automatically adjust the white balance during capture. The ambient light sensor 180L may further cooperate with the optical proximity sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touch.

Fingerprint sensor 180H is configured to capture a fingerprint. Electronic device 100 may use the captured fingerprint characteristics to perform fingerprint-based unlocking, application lock access, fingerprint-based photographing, fingerprint-based call answering, etc.

The temperature sensor 180J is configured to detect a temperature. In some embodiments, the electronic device 100 executes a temperature handling policy based on the temperature detected by the temperature sensor 180J. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In some other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid an abnormal shutdown of the electronic device 100 caused by a low temperature. In some other embodiments, when the temperature is lower than yet another threshold, the electronic device 100 boosts the output voltage of the battery 142 to avoid an abnormal shutdown caused by a low temperature.

The touch sensor 180K may be referred to as a "touch panel." The touch sensor 180K may be disposed on the display 194, with the touch sensor 180K and the display 194 forming a touch screen, also referred to as a "touch screen." The touch sensor 180K is configured to detect touch operations performed on or near the touch sensor. The touch sensor may forward the detected touch operations to an application processor to determine a type of touch event. A visual output associated with the touch operation may be provided on the display 194. In some other embodiments, the touch sensor 180K may alternatively be disposed on the surface of the electronic device 100 at a location different from the location of the display 194.

The bone conduction sensor 180M may acquire a vibration signal. In some embodiments, the bone conduction sensor 180M may acquire a vibration signal from a sound-vibrating bone of the human body. The bone conduction sensor 180M may also receive a blood pressure pulsation signal by contacting the pulse of the human body. In some embodiments, the bone conduction sensor 180M may be disposed in a headset so as to be integrated into a bone conduction headset. The audio module 170 may analyze an audio signal based on the vibration signal of the sound-vibrating bone acquired by the bone conduction sensor 180M to implement a voice function. The application processor may analyze heart rate information based on the blood pressure pulsation signal acquired by the bone conduction sensor 180M to implement a heart rate detection function.

Buttons 190 include a power button, a volume button, etc. Buttons 190 may be mechanical or touch-sensitive buttons. Electronic device 100 may receive button input and generate button signal input associated with user settings and function control of electronic device 100.

The motor 191 may generate a vibration alert. The motor 191 may be used for vibration alerts for incoming calls and may also be used for touch vibration feedback. For example, touch operations on different applications (e.g., taking pictures and playing audio) may correspond to different vibration feedback effects. For touch operations on different areas of the display 194, the motor 191 may also generate correspondingly different vibration feedback effects. Different application scenarios (such as time reminders, information reception, alarm clocks, and games) may also correspond to different vibration feedback effects. The touch vibration feedback effects may be further customized.

Indicator 192 may be an indicator and may be configured to indicate charging status and power changes, or may be configured to indicate messages, missed calls, notifications, etc.

The SIM card interface 195 is configured to connect to a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to effect contact or separation with the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support a nano-SIM card, a micro-SIM card, a SIM card, etc. Multiple cards can be inserted into the same SIM card interface 195. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with an external memory card. The electronic device 100 interacts with a network through the SIM card to implement functions such as calling and data communication. In some embodiments, electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card may be embedded in electronic device 100 and cannot be separated from electronic device 100.

It should be understood that the electronic device 100 shown in FIG. 9 is only an example, and that the electronic device 100 may have more or fewer components than those shown in FIG. 9, may combine two or more components, or may have a different component configuration. The various components shown in FIG. 9 may be implemented using hardware, software, or a combination of hardware and software, including one or more signal processors and/or application specific integrated circuits.

The following describes the software structure of the electronic device 100 according to one embodiment of the present application.

Figure 10 shows an example of the software structure of electronic device 100 according to one embodiment of the present application.

As shown in FIG. 10, the software system of the electronic device 100 may use a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In the following, the software structure of the electronic device 100 is described using an example.

The software is divided into layers by using a layered architecture, with each layer having a distinct role and task. These layers communicate with each other by using software interface connections. In some embodiments, the software structure of electronic device 100 is divided into three layers from top to bottom: application layer, application framework layer, and kernel layer.

The application layer may include a set of application packages.

As shown in FIG. 10, the application package may include applications such as camera, gallery, calendar, phone, maps, navigation, WLAN, Bluetooth, music, video, and messaging. Video may be a video application as mentioned in the embodiments of this application.

The application framework layer provides an application programming interface (API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functionality.

As shown in FIG. 10, the application framework layer may include a window manager, a content provider, a view system, a telephone manager, a resource manager, a notification manager, a video processing system, etc.

A window manager is configured to manage window programs. The window manager may get the display size, determine if there is a status bar, lock the screen, take screenshots, etc.

Content providers are configured to store and retrieve data and make this data accessible to applications. Data can include video, images, audio, calls made and received, browsing history and bookmarks, address books, etc.

The view system includes visual controls, such as a control for displaying text and a control for displaying pictures. The view system can be configured to build applications. A display interface can include one or more views. For example, a display interface that includes a message notification icon can include a view for displaying text and a view for displaying pictures.

The phone manager is configured to provide management of communication functions of the electronic device 100, such as call status (including answering, rejecting, etc.).

The resource manager provides resources for an application, such as localized strings, icons, images, layout files, and video files.

The notification manager may be configured to allow applications to display notification information in the status bar and to communicate notification messages. The notification manager may disappear automatically after a short pause without requiring user interaction. For example, the notification manager may be configured to notify of download completion, provide message reminders, etc. The notification manager may alternatively be a notification that appears in the status bar at the top of the system in the form of a graph or scrollbar text, such as a notification about an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, textual information may be displayed in the status bar, an announcement may be made, the electronic device may vibrate, an indicator light may flash, etc.

The video processing system may be configured to execute the caption display method provided in the embodiments of the present application. The video processing system may include a caption decoding module, a video frame gamut interpretation module, a video frame synthesis module, a video frame queue, and a video rendering module. For specific functions of each module, please refer to the relevant contents in the above-mentioned embodiments. Details will not be described again here.

The kernel layer is a layer between the hardware and the software. The kernel layer includes at least a display driver, a camera driver, a Bluetooth driver, and a sensor driver.

Below, an example of the operating process of the software and hardware of the electronic device 100 is described with reference to a shooting scenario.

When the touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into an original input event (including information such as the touch coordinates and timestamp of the touch operation). The original input event is stored in the kernel layer. The application framework layer obtains the original input event from the kernel layer and identifies a control corresponding to the input event. For example, the touch operation is a single tap operation, and the control corresponding to the single tap operation is a control of a camera application icon. The camera application calls an interface in the application framework layer to start the camera application. Then, it calls the kernel layer to start the camera driver and use the camera 193 to take still images or videos.

The following describes the structure of another electronic device 100 according to one embodiment of the present application.

Figure 11 shows an example of a configuration of another electronic device 100 according to one embodiment of the present application.

As shown in FIG. 11, the electronic device 100 may include a video application 1100 and a video processing system 1110.

Video application 1100 may be a system application installed on electronic device 100 (e.g., the "Video" application shown in FIG. 2A) or may be an application provided by a third party and installed on electronic device 100 and having video playback capabilities. Video application 1100 is primarily configured to play videos.

The video processing system 1110 may include a video decoding module 1111, a caption decoding module 1112, a video frame gamut interpretation module 1113, a video frame synthesis module 1114, a video frame queue 1115, and a video rendering module 1116.

The video decoding module 1111 may receive the video information stream transmitted by the video application 1100 and decode the video information stream to generate video frames.

The caption decoding module 1112 may receive a caption information stream transmitted by the video application 1100, decode the caption information stream to generate caption frames, and transmit the caption frames with a mask based on the mask parameters transmitted by the video frame gamut interpretation module 1113 to enhance caption recognition.

The video frame color gamut interpretation module 1113 may analyze the caption recognizability to generate a caption recognizability analysis result, and calculate mask parameters (mask color value and transparency) corresponding to the caption based on the caption recognizability analysis result.

The video frame synthesis module 1114 may overlay and combine the video frames and caption frames to generate the video frames to be displayed.

The video frame queue 1115 may store the video frames to be displayed that are sent by the video frame composition module 1114.

The video rendering module 1116 may render the video frames to be displayed based on a time sequence to generate rendered video frames and send the rendered video frames to the video application 1100 for video playback.

For further details regarding the function and working principle of the electronic device 100, please refer to the relevant contents in the above-mentioned embodiment. The details will not be described again here.

It should be understood that the electronic device 100 shown in FIG. 11 is only an example, and that the electronic device 100 may have more or fewer components than those shown in FIG. 11, may combine two or more components, or may have a different component configuration. The various components shown in FIG. 11 may be implemented in hardware, software, or a combination of hardware and software.

The aforementioned modules may be divided by functionality. In an actual product, the modules may be different functions performed by the same software module.

The following describes the structure of another electronic device 100 according to one embodiment of the present application.

Figure 12 shows an example of a configuration of another electronic device 100 according to one embodiment of the present application.

As shown in FIG. 12, the electronic device 100 may include a video application 1200. The video application 1200 may include a video decoding module 1211, a caption decoding module 1212, a video frame gamut interpretation module 1213, a video frame composition module 1214, a video frame queue 1215, and a video rendering module 1216.

Video application 1200 may be a system application installed on electronic device 100 (e.g., the "Video" application shown in FIG. 2A) or may be an application provided by a third party and installed on electronic device 100 and having video playback capabilities. Video application 1200 is primarily configured to play videos.

The acquisition and display module 1210 may acquire the video information stream and the caption information stream and display rendered video frames transmitted by the video rendering module 1216, etc.

The video decoding module 1211 may receive the video information stream transmitted by the acquisition and display module 1210 and decode the video information stream to generate video frames.

The caption decoding module 1212 receives the caption information stream transmitted by the acquisition and display module 1210, decodes the caption information stream to generate caption frames, and may generate caption frames with a mask based on the mask parameters transmitted by the video frame gamut interpretation module 1213 to enhance caption recognition.

The video frame color gamut interpretation module 1213 may analyze the caption recognizability to generate a caption recognizability analysis result, and calculate mask parameters (mask color value and transparency) corresponding to the caption based on the caption recognizability analysis result.

The video frame synthesis module 1214 may overlay and combine the video frames and caption frames to generate the video frames to be displayed.

The video frame queue 1215 may store the video frames to be displayed that are sent by the video frame composition module 1214.

The video rendering module 1216 may render the video frames to be displayed based on a time sequence to generate rendered video frames and transmit the rendered video frames to the acquisition and display module 1210 for video playback.

For further details regarding the function and working principle of the electronic device 100, please refer to the relevant contents in the above-mentioned embodiment. The details will not be described again here.

It should be understood that the electronic device 100 shown in FIG. 12 is only one example, and that the electronic device 100 may have more or fewer components than those shown in FIG. 12, may combine two or more components, or may have a different component configuration. The various components shown in FIG. 12 may be implemented in hardware, software, or a combination of hardware and software.

The above embodiments are only intended to describe the technical solutions of the present application, and are not intended to limit the present application. Although the present application has been described in detail with reference to the above embodiments, it should be understood that those skilled in the art can further modify the technical solutions described in the above embodiments, or make equivalent substitutions for some technical features thereof, without departing from the scope of the technical solutions of the embodiments of the present application.

Claims (15)

A caption display method, comprising:
the electronic device playing the first video;
When the electronic device displays a first interface, the first interface includes a first picture and a first caption, the first caption is displayed on a first region of the first picture in a floating manner that overlaps the first caption on the first picture by using a first mask as a background, the first region is a region in the first picture that corresponds to a display position of the first caption, and a difference value between a color value of the first caption and a color value of the first region is a first value;
When the electronic device displays a second interface different from the first interface after displaying the first interface, the second interface includes a second picture and the first caption, no mask is displayed for the first caption, the first caption is displayed on a second region of the second picture in a floating manner that overlays the first caption on the second picture , the second region is a region in the second picture that corresponds to the display position of the first caption, a difference value between the color value of the first caption and the color value of the second region is a second value, the second value is greater than the first value;
the first picture is one picture in the first video and the second picture is another picture in the first video;
a difference value between the color value of the first mask and the color value of the first caption is greater than the first value;
method. Before the electronic device displays the first interface, the method further comprises:
The electronic device obtains a first video file and a first caption file, the first video file and the first caption file carrying the same time information;
generating, by the electronic device, a first video frame based on the first video file, the first video frame being used to generate the first picture;
the electronic device generating a first caption frame based on the first caption file and obtaining the color value and the display position of the first caption from the first caption frame, wherein time information carried in the first caption frame is the same as time information carried in the first video frame;
determining, by the electronic device, the first region based on the display position of the first caption;
generating, by the electronic device, the first mask based on the color values of the first caption or the color values of the first region;
2. The method of claim 1, further comprising: the electronic device overlaying the first caption on the first mask in the first caption frame to generate a second caption frame, and combining the second caption frame with the first video frame. Prior to the step of the electronic device generating the first mask based on the color values of the first caption or the color values of the first region, the method further comprises:
The method of claim 2 , further comprising: the electronic device determining that the first value is less than a first threshold value. Specifically, the step of the electronic device determining that the first value is less than a first threshold value includes:
the electronic device dividing the first region into N first sub-regions, where N is a positive integer;
and determining, by the electronic device, based on the color value of the first caption and the color values of the N first sub-regions, that the first value is less than the first threshold. Specifically, the step of generating the first mask based on the color value of the first caption or the color value of the first region by the electronic device includes:
determining, by the electronic device, a color value of the first mask based on the color value of the first caption or the color values of the N first sub-regions;
and the electronic device generating the first mask based on the color values of the first mask. Specifically, the step of the electronic device determining that the first value is less than a first threshold value includes:
the electronic device dividing the first region into N first sub-regions, where N is a positive integer;
determining whether to combine adjacent first sub-regions into a second sub-region based on a difference value between color values of the adjacent first sub-regions;
when the difference value between the color values of the adjacent first sub-regions is less than a second threshold, the electronic device merges the adjacent first sub-regions into the second sub-region;
and determining, by the electronic device, based on the color value of the first caption and the color value of the second sub-region, that the first value is less than the first threshold. The method of claim 6, wherein the first region includes M second subregions, where M is a positive integer and is less than or equal to N, the second subregion includes one or more first subregions, and the number of first subregions included in each second subregion is the same as or different from the number of first subregions included in another second subregion. Specifically, the step of generating the first mask based on the color value of the first caption or the color value of the first region by the electronic device includes:
The electronic device sequentially calculates color values of M first sub-masks, which are masks corresponding to the M second sub-regions, based on the color value of the first caption or the color values of the M second sub-regions;
generating, by the electronic device, M first sub-masks based on the color values of the M first sub-masks, the M first sub-masks being combined into the first mask ;
The method of claim 7, comprising: The method comprises:
When the electronic device displays a third interface, the third interface includes a third picture and the first caption, the first caption includes at least a first portion and a second portion, a second submask is displayed for the first portion, a third submask is displayed or the third submask is not displayed for the second portion, and color values of the second submask are different from color values of the third submask.
The method of claim 1 further comprising: The method of claim 1, wherein the display position of the first mask is determined based on the display position of the first caption. The method of claim 1, wherein in the first picture and the second picture, the display position of the first caption relative to a display screen of the electronic device is not fixed or is fixed, and the first caption is a segment of a character or symbol that is displayed continuously. Before the electronic device displays the first interface, the method further comprises:
The method of claim 1 , further comprising: the electronic device setting the transparency of the first mask to less than 100%. Before the electronic device displays the second interface, the method further comprises:
a step of generating a second mask based on the color values of the first caption or the color values of the second region by the electronic device, and overlaying the first caption on the second mask, the color values of the second mask being preset color values and the transparency of the second mask being 100%;
Or,
The method of claim 1 , further comprising: the electronic device skipping generation of the second mask. 14. An electronic device comprising one or more processors and one or more memories coupled to the one or more processors, the one or more memories configured to store computer program code comprising computer instructions, the one or more processors, when executed by the one or more processors, enabling the electronic device to perform a method according to any one of claims 1 to 13 . A computer program comprising program instructions which, when executed on an electronic device, enable the electronic device to carry out the method according to any one of claims 1 to 13 .

Images