JP7465486B2 - Systems, methods, and computer-readable media for image recognition - Google Patents

Description

The present invention relates to image or video processing in live video streaming or video teleconferencing.

Various technologies are known that allow users to participate in online communications with each other. Applications include live streaming, live teleconferencing, etc. With the popularity of these applications, there is an increasing demand from users for an improved interaction experience and smoother synchronization during communication.

A method according to one embodiment of the present invention is an image recognition method. The method of the present invention includes the steps of obtaining a first pattern to be displayed on a user terminal, comparing the first pattern with a portion of a user displayed on the user terminal, and updating a result of the comparison.

A system according to one embodiment of the present invention is a system for image recognition that includes one or more processors that execute machine-readable instructions to perform the steps of obtaining a first pattern for display on a user terminal, comparing the first pattern with a portion of a user displayed on the user terminal, and updating a result of the comparison.

A computer-readable medium according to one embodiment of the present invention is a non-transitory computer-readable medium including a program for image recognition, the program causing one or more computers to execute the steps of obtaining a first pattern displayed on a user terminal, comparing the first pattern with a portion of the user displayed on the user terminal, and updating the results of the comparison.

FIG. 2 is a schematic diagram illustrating an example of a group call. 1 is a schematic diagram illustrating an example interaction in a group call according to some embodiments of the present invention. 1 is a schematic diagram illustrating an example interaction in a group call according to some embodiments of the present invention. 1 is a schematic diagram illustrating an example interaction in a group call according to some embodiments of the present invention. 1 is a schematic diagram illustrating an example interaction in a group call according to some embodiments of the present invention. 1 is a schematic diagram illustrating an example interaction in a group call according to some embodiments of the present invention. 1 is a schematic diagram illustrating an example interaction in a group call according to some embodiments of the present invention. 1 is a schematic diagram illustrating an example interaction in a group call according to some embodiments of the present invention. FIG. 2 is a schematic diagram illustrating an exemplary interaction in live streaming according to some embodiments of the present invention. 1 is a schematic diagram illustrating the configuration of a communication system according to some embodiments of the present invention. 2 is an exemplary sequence diagram illustrating the operation of a communication system in accordance with some embodiments of the present invention. 1 is an exemplary functional block diagram of a communication system in accordance with some embodiments of the present invention;

Some live streaming services, applications (APPs) or platforms allow multiple users (distributors, viewers, broadcasters, anchors, etc.) to participate in a group call mode or a conference call mode, in which the images of multiple users are displayed simultaneously on the screen of a user terminal to indicate the group call or participation in the group call. The user terminal can be a smartphone, tablet, PC, or laptop, and is used by one of the users to participate in the group call.

Figure 1 shows a schematic diagram of an example of a group call. S1 is a screen of a user terminal that displays the group call. RA is a display area in the screen S1 that displays a live video of user A. RB is a display area in the screen S1 that displays a live video of user B. RC is a display area in the screen S1 that displays a live video of user C. RD is a display area in the screen S1 that displays a live video of user D. The live video of user A may be captured and provided by a video capture device such as a camera placed near user A. The live video of user B may be captured and provided by a video capture device such as a camera placed near user B. The live video of user C may be captured and provided by a video capture device such as a camera placed near user C. The live video of user D may be captured and provided by a video capture device such as a camera placed near user D.

Conventionally, users A, B, C, and D act in their own display areas, and there is a lack of collective interaction across different display areas. To increase users' motivation to participate in group calls, more collective or interesting interactions are desired.

Conventionally, when a delay, lag, out-of-sync, or other problem occurs in the connection between users A, B, C, and D during a group call, one user may not immediately notice the connection problem and continue talking while other users are not able to receive their messages (audio and video) in sync or smoothly. Or, even if there is a concern or suspicion that an out-of-sync problem is occurring, the user has no convenient way to test or uncover it. Therefore, it is desirable to enable a user to initiate a convenient way to test the sync status of a group call connection.

2A, 2B, and 2C are schematic diagrams illustrating exemplary interactions in a group call according to some embodiments of the present invention.

As shown in FIG. 2A, a trajectory or contour O1 is displayed on the screen S1. The contour O1 is displayed across the display areas RA, RB, RC, and RD. A portion of the contour O1 is displayed in each display area. In this embodiment, the display areas RA, RB, RC, and RD are separated from one another. In some embodiments, the display areas RA, RB, RC, and RD may overlap. In this embodiment, the contour O1 is heart-shaped. In some embodiments, the contour may be any shape, such as a circle, an oval, or a rectangle. The contour O1 may be referred to as a first pattern. The contour O1 may be obtained from a server or a user terminal.

As shown in FIG. 2B, users A, B, C, and D attempt to match the contour O1. The contour O1 is compared to portions of users A, B, C, and D. User A uses his hands and arms (which may be partially covered by clothing) to match or adapt to the portion of the contour O1 displayed in the display area RA. User B uses his hands and arms (which may be partially covered by clothing) to match or adapt to the portion of the contour O1 displayed in the display area RB. User C uses his hands and arms (which may be partially covered by clothing) to match or adapt to the portion of the contour O1 displayed in the display area RC. User D uses his hands and arms (which may be partially covered by clothing) to match or adapt to the portion of the contour O1 displayed in the display area RD.

As shown in FIG. 2C, a special effect O11 is displayed on the screen S1. The special effect O11 can function as an update of the comparison result between the contour O1 and the parts of user A, user B, user C, user D. The special effect O11 can be a graphic object, an animation object, or an embodiment object of the contour O1. In some embodiments, the special effect O11 is displayed when a collective shape or collective pattern formed or configured by the parts of the users (such as body parts or non-body parts of users A, B, C, D) matches or fits the contour O1. The collective pattern is formed across the display areas RA, RB, RC, RD.

In some embodiments, the collective pattern formed by the users may be recognized by a user terminal or a system providing the group call service having a pattern recognition process. In some embodiments, the pattern recognition process may include a gesture recognition process, a skin recognition process, a contour recognition process, a shape recognition process, or an object recognition process. In some embodiments, the pattern recognition process may include an image comparison process (such as a comparison of a series of images with a predefined initial background image in each user's display area) or a moving object detection process. In some embodiments, a motion estimation (ME) method or a motion compensation (MC) method may be used. In some embodiments, the collective pattern may be referred to as a second pattern.

Determining whether the aggregate pattern matches the contour O1 (or comparing the contour O1 with a portion of the user) may be performed by an image or pattern comparison/matching process, which may include calculating a similarity coefficient (such as a correlation value) between the aggregate pattern and the contour O1. In some embodiments, calculating the similarity coefficient may include a correlation calculation process, a trajectory superposition process, a normalization process, or a minimum distance determination process. The image or pattern comparison process may be performed by a user terminal or a system providing the group call service. In some embodiments, the special effect O11 is displayed if the similarity coefficient is equal to or greater than a predetermined value.

3A and 3B show another schematic diagram illustrating an exemplary interaction in a group call according to some embodiments of the present invention.

As shown in FIG. 3A, the contour O1 continues to move with an offset, adjusting the position of the user's part (such as the hand) in real time. In this embodiment, the contour O1 is moved horizontally. In another embodiment, the contour O1 may be moved vertically, diagonally, or in any direction. In some embodiments, the contour O1 may be moved forwards and backwards. In some embodiments, the contour O1 may be moved periodically with a time period such as 1 second, 2 seconds, or 3 seconds. In some embodiments, the contour O1 may be moved intermittently with different time periods.

As shown in FIG. 3B, a special effect O11 is displayed on the screen S1. The special effect O11 may be a graphic object, an animation object, or an embodiment object of the contour O1. In some embodiments, the special effect O11 is displayed when a collective shape or collective pattern formed or configured by the parts of the users (such as body parts or non-body parts of users A, B, C, D) matches or fits the contour O1. In this embodiment, the contour O1 is a moving contour or moving body. Therefore, users A, B, C, D need to move their parts in accordance with the movement of the contour O1 so that their collective pattern matches or fits the contour O1 continuously or periodically.

The aggregate pattern may be continuously or periodically recognized by the user terminal or a system providing the group call service having a pattern recognition process. The determination of whether the aggregate pattern matches the contour O1 may be performed by an image or pattern comparison process, which may include continuously or periodically calculating a similarity coefficient (such as a correlation value) between the moving aggregate pattern and the moving contour O1. The image or pattern comparison process may be performed by the user terminal or a system providing the group call service. In some embodiments, the special effect O11 is displayed when the similarity coefficient is continuously or periodically equal to or greater than a predetermined value. For example, the special effect O11 is displayed when the similarity coefficient is equal to or greater than a predetermined value for a predetermined period (such as 5 seconds or 10 seconds) or a predetermined number of cycles (such as the number of times the contour O1 moves 3 times or 5 times).

Figure 4 shows another schematic diagram illustrating exemplary interactions in a group call according to some embodiments of the present invention.

As shown in FIG. 4, the contour O1 may be rotated continuously, periodically, or intermittently at an offset angle. In some embodiments, the contour O1 may be rotated back and forth. In some embodiments, the contour O1 may be rotated periodically with a time period such as 1 second, 2 seconds, or 3 seconds. In some embodiments, the contour O1 may be rotated intermittently with a different time period.

The special effect O11 is displayed on the screen S1. The special effect O11 may be a graphic object, an animation object, or an embodiment object of the contour O1. In some embodiments, the special effect O11 is displayed when a collective shape or collective pattern formed or configured by the parts of the users (such as body parts or non-body parts of users A, B, C, D) matches or fits the contour O1. In this embodiment, the contour O1 is a moving contour or moving body (here "moving" includes "rotating"). Thus, users A, B, C, D need to move their parts in accordance with the movement (or rotation) of the contour O1 so that their collective pattern matches or fits the contour O1 continuously or periodically.

The aggregate pattern may be continuously or periodically recognized by the user terminal or a system providing the group call service having a pattern recognition process. The determination of whether the aggregate pattern matches the contour O1 may be performed by an image or pattern comparison process, which may include continuously or periodically calculating a similarity coefficient (such as a correlation value) between the moving/rotating aggregate pattern and the moving/rotating contour O1. The image or pattern comparison process may be performed by the user terminal or a system providing the group call service. In some embodiments, the special effect O11 is displayed when the similarity coefficient is equal to or greater than a predetermined value continuously or periodically. For example, the special effect O11 is displayed when the similarity coefficient is equal to or greater than a predetermined value for a predetermined period (such as 5 seconds or 10 seconds) or a predetermined number of cycles (such as the number of times the contour O1 moves 3 times or 5 times).

In some embodiments, the continuous or periodic matching of the contour O1 with the aggregate pattern formed by the users can also be used as a way to test the synchronization state or synchronization level of the connection between the users of the group call. For example, if a user feels that communication is not smooth, the user can initiate a series of steps through the user terminal, such as displaying the contour O1, periodically moving the contour O1, periodically recognizing the aggregate pattern formed by the user, and periodically comparing the aggregate pattern with the contour O1. If periodic matching (e.g., matching for a predetermined period or a predetermined number of cycles) is achieved, the synchronization level may be determined to be acceptable.

For example, if the similarity coefficient between the contour O1 and the set pattern is equal to or greater than a predetermined value for a predetermined period of time, a message may be displayed on the user terminal indicating that the synchronization level is acceptable. The message may be a special effect, such as special effect O11, or any other form. Conversely, if the similarity coefficient between the contour O1 and the set pattern is less than a predetermined value, a message may be displayed on the user terminal indicating that the synchronization level is unacceptable. In that case, the user may know that the connection is not good and may slow down the pace of communication or change to another communication method. In some embodiments, the above-mentioned synchronization test may be performed before starting an online game or any event that requires a smooth connection between users.

Figure 5 shows another schematic diagram illustrating exemplary interactions in a group call according to some embodiments of the present invention.

In this embodiment, users A, B, C, and D use an object or tool T1 to match or conform to the contour O1. The tool T1 may comprise a flexible material and may be bent or deformed to conform to a particular shape. For example, user A may bend the tool T1 to conform to the portion of the contour O1 displayed in the display area RA. In some embodiments, information such as the color or shape of the tool T1 may be taught to a user terminal or a system providing the group call service. The user terminal or the system may use the information of the tool T1 to recognize the collective pattern formed by users A, B, C, and D.

Figure 6 shows another schematic diagram illustrating an example interaction during live streaming (which may be in the form of a group call or in a form other than a group call) according to some embodiments of the present invention.

In this embodiment, users A, B, C, D are displayed within a single display area on the screen S1 of the user terminal. Users A, B, C, D may actually be in the same space. Alternatively, users A, B, C, D may be in separate spaces and their respective images or videos may be combined within one display area through image or video processing performed by the user terminal or a system participating in or providing the streaming service. As in the previous embodiment, users A, B, C, D attempt to form a collective pattern that matches the contour O1. If a match is achieved, special effects may occur.

In some embodiments, the reward may be a special effect for successful matching of the aggregate pattern (formed by the user's portion) with a predefined contour. The matching mechanism may also be considered as a way to enhance the atmosphere and interaction of live streaming involving multiple users. For example, successful matching may initiate or trigger a donation or a gift from the user who initiated the matching process. The user who initiated the matching process may be a streamer (or anchor, broadcaster) or a viewer (or fan) of the stream. Thus, a more collective or entertaining interaction between users involved in streaming or live broadcasting is realized. This may lead to an increase in gift sending, which may increase the revenue of the streaming service provider or streamer, and may further lead to improved platform performance and content production. In some embodiments, the sending of a gift is triggered or realized based on the result of comparing the contour O1 with the portion of the user.

FIG. 7 is a schematic diagram showing the configuration of a communication system according to some embodiments of the present invention. The communication system 1 can provide a live streaming service with interaction through content. The "content" here refers to digital content that can be played on a computer device. The communication system 1 allows users to participate in real-time exchanges with other users online. The communication system 1 includes a plurality of user terminals 10, a back-end server 30, and a streaming server 40. The user terminals 10, the back-end server 30, and the streaming server 40 are connected via a network 90 (which may be, for example, the Internet). The back-end server 30 can be a server that synchronizes interactions between the user terminals and/or the streaming server 40. In some embodiments, the back-end server 30 can be an origin server of an application (APP) provider. The streaming server 40 is a server for handling or providing streaming data or video data. In some embodiments, the back-end server 30 and the streaming server 40 can be independent servers. In some embodiments, the back-end server 30 and the streaming server 40 can be integrated into one server. In some embodiments, the user terminal 10 is a client device for live streaming. In some embodiments, the user terminal 10 may be referred to as a viewer, streamer, anchor, podcaster, audience, listener, etc. Each of the user terminal 10, the backend server 30, and the streaming server 40 is an example of an information processing device. In some embodiments, the streaming may be live streaming or video playback. In some embodiments, the streaming may be audio streaming and/or video streaming. In some embodiments, the streaming may include content such as online shopping, talk shows, talent shows, entertainment events, sporting events, music videos, movies, comedy shows, concerts, group calls, conference calls, etc.

Figure 8 shows an exemplary sequence diagram illustrating the operation of a communication system based on some embodiments of the present invention.

The processing unit set 120 may include components or devices of the user terminal 10A used by user A. For example, the processing unit set 120 may include a renderer, an encoder, a decoder, a CPU, a GPU, a controller, a processor, and/or an image/video capture device such as a camera.

The display 122 may refer to the display of the user terminal 10A used by user A.

The processing unit set 140 may include components or devices of the user terminal 10B used by user B. For example, the processing unit set 140 may include a renderer, an encoder, a decoder, a CPU, a GPU, a controller, a processor, and/or an image/video capture device such as a camera.

The display 142 may refer to the display of the user terminal used by user B.

The message unit 32 is configured to communicate messages or signals with devices such as a user terminal, a back-end server, or a streaming server. In some embodiments, the message unit 32 may be implemented within a back-end server. In some embodiments, the message unit 32 may be implemented independently of a back-end server.

The shape recognition unit 50 is configured to perform a shape/pattern detection/recognition process, including a gesture recognition process, a skin recognition process, a contour recognition process, and/or an object recognition process. The shape recognition unit 50 may be configured to perform a pattern comparison process, including a process of calculating a similarity coefficient or a correlation value between two patterns. According to another embodiment, the shape recognition unit 50 may be implemented in a user terminal, a back-end server, or independently. In this embodiment, both the pattern recognition process and the pattern comparison process are performed by the shape recognition unit 50. In some embodiments, these two processes may be performed by different units, and may be implemented in a user terminal, a back-end server, or independently based on the actual application.

In step S800, the processing unit set 120 renders video data for display on the display 122, which may include video of user A captured by a video capture device.

In step S802, the processing unit set 120 transmits the encoded video data to the streaming server 40. The encoded video data may include a video of user A encoded by an encoder.

In step S804, the processing unit set 140 renders video data for display on the display 142, which may include video of user B captured by a video capture device.

In step S806, the processing unit set 140 transmits the encoded video data to the streaming server 40. The encoded video data may include a video of user B encoded by an encoder.

In step S808, the video data of user B is retrieved from the streaming server 40 and displayed on the display 122. Note that some processing such as decoding, processing, and rendering of the retrieved video data is omitted here.

In step S810, the video data of user A is retrieved from the streaming server 40 and displayed on the display 142. Note that some processing such as decoding, processing, and rendering of the retrieved video data is omitted here.

In step S812, the shape or pattern information is transmitted from the message unit 32 to the processing unit set 120.

In step S814, the shape or pattern information is sent from the message unit 32 to the processing unit set 140. Steps S812 and S814 may be performed simultaneously.

In some embodiments, the transmission of the shape information may be triggered by a user action via a user terminal. The user may be a streamer, broadcaster, anchor, or viewer. In some embodiments, the transmission of the shape information may be triggered by a communications or streaming service provider.

In step S816, the processing unit set 120 renders a shape or pattern corresponding to the received or acquired shape information and displays it on the display 122. Thus, User A can see on the screen the pattern that needs to be matched (e.g., the portion of the pattern in User A's display area).

In step S818, the processing unit set 140 renders a shape or pattern corresponding to the received or acquired shape information and displays it on the display 142. Thus, user B can see on the screen the pattern that needs to be matched (e.g., the portion of the pattern in user B's display area).

In step S820, User A attempts to fill, fit, or conform to the shape. For example, User A attempts to behave in such a way that portions of User A or a tool used by User A (as displayed on the display 122) conform to the portions of the shape displayed in User A's display area.

In step S822, the user attempts to fill, fit, or conform to the shape. For example, user B attempts to behave in such a way that portions of user B or a tool used by user B (as displayed on the display 142) conform to the portions of the shape displayed in user B's display area.

In step S824, the shape recognition unit 50 detects or recognizes a collection shape or collection pattern formed by user A and user B. The recognized collection shape may be displayed on the display 122. In this embodiment, the shape recognition unit 50 compares the collection shape with an initial shape (or a predetermined shape) displayed on the display 122 in response to the shape information received in step S812 (e.g., in step S816). The shape recognition unit 50 may calculate a similarity coefficient or correlation value between the collection shape and the predetermined shape, and determine whether matching is established at a predetermined threshold.

In step S828, the matching result is transmitted from the shape recognition unit 50 to the message unit 32.

In step S830, the message unit 32 notifies the processing unit set 120 of the matching result.

In step S832, the message unit 32 notifies the processing unit set 140 of the matching result. Steps S830 and S832 may be performed simultaneously.

In step S834, if the match is successful, a special effect is rendered on the display 122. The special effect may include a message indicating that communication between user A and user B has been smoothly synchronized.

In step S836, if the match is successful, a special effect is rendered on the display 142. The special effect may include a message indicating that communication between user A and user B has been smoothly synchronized.

In some embodiments, the message unit 32 periodically transmits shape information to each user terminal, and the shape information can include an offset each time. Thus, a moving or rotating shape can be displayed on the display of each user terminal. Subsequent pattern recognition and pattern matching processes can also be performed periodically and used as a synchronization level check for communication. For example, a user can visually know the state of synchronization by looking at the display.

Regarding the screen S1 in FIG. 1 (or FIG. 2A), the screen S1 may be implemented on a user terminal of any one of user A, user B, user C, and user D. In some embodiments, the communication system may be configured to swap the display areas of any two of the users. The communication system may receive an instruction to swap from one of the user terminals. This allows for greater freedom of arrangement and further improves interaction between users.

In some embodiments, the screen S1 of FIG. 1 (or FIG. 2A) may be implemented on a streamer's user terminal. In this case, the streamer instructs users A-D on what to do, deepening the interaction between the two. This strengthens the sense of unity between the streamer and users A-D. Any of users A-D can be a streamer or a viewer.

In some embodiments, the trajectory or contour O1 is selected by one of the users or voted for from a list of candidate contours, each of which is associated with a respective gift amount or with an amount of value associated with its respective gift. The amount (or amount of value) may be set such that the higher the amount, the more difficult it is to match the respective contour. In some embodiments, the value of the gift corresponds to the difficulty of matching the contour/pattern. Alternatively, the contour O1 may be selected randomly from a list of candidate contours or the contour O1 may be determined based on the profiles of the participating users A-D.

FIG. 9 shows an exemplary functional block diagram of a communication system according to some embodiments of the present invention. In this embodiment, the shape recognition unit 50 may be implemented independently, not in the backend server 30, the streaming server 40, or the user terminal. In another embodiment, the shape recognition unit 50 may be implemented in the backend server, the streaming server, the cache server, or the user terminal. In this embodiment, the message unit 32 is implemented in the backend server 30. In another embodiment, the message unit 32 may be implemented outside the backend server 30. For example, the message unit 32 may be implemented independently, or in the streaming server, the cache server, or the user terminal.

The user terminal 10A is a user terminal used by user A. The user terminal 10A includes a processing unit set 120 and a display 122. The processing unit set 120 may include a renderer, an encoder, a decoder, a CPU, a GPU, a controller, a processor, and/or an image/video capture device such as a camera.

User terminal 10B is a user terminal used by user B. The user terminal 10B includes a processing unit set 140 and a display 142. The processing unit set 120 may include a renderer, an encoder, a decoder, a CPU, a GPU, a controller, a processor, and/or an image/video capture device such as a camera.

8 and 9, the message unit 32 may transmit shape or pattern information to the processing unit set 120 of the user terminal 10A and the processing unit set 140 of the user terminal 10B. The transmission of the above-mentioned pattern information may be triggered by a user terminal (such as a streamer's or viewer's user terminal), a server in the system, or an application supporting communication, in which the processing unit set 120 renders a shape or pattern corresponding to the received or acquired shape information and displays it on the display 122. Thus, user A can check the pattern that needs to be matched (e.g., the part of the pattern in user A's display area) on the screen.

The processing unit set 140 renders a shape or pattern corresponding to the received or acquired shape information and displays it on the display 142. Thus, user B can see on the screen the pattern that needs to be matched (e.g., the portion of the pattern in user B's display area).

Then, the processing unit set 120 and the processing unit set 140 collect video/image data of the parts of the pattern that user A and user B are trying to match, and send the data to the shape recognition unit 50.

The shape recognition unit 50 detects or recognizes a collection shape or collection pattern formed by user A and user B. In this embodiment, the shape recognition unit 50 compares the collection shape with an initial shape (or a predetermined shape) displayed on the display 122 in response to the shape information received in step S812 of FIG. 8 (e.g., in step S816 of FIG. 8). The shape recognition unit 50 may calculate a similarity coefficient or correlation value between the collection shape and the predetermined shape, and determine whether a match is established at a predetermined threshold. The shape recognition unit 50 then transmits the matching result to the message unit 32.

The message unit 32 notifies the processing unit set 120 and the processing unit set 140 of the matching result. If the matching is successful, the processing unit set 120 and the processing unit set 140 may render special effects for display on the display 122 and the display 142. The special effects may include a message indicating that communication between user A and user B has been smoothly synchronized.

The present invention improves interaction during conference calls or group calls, promotes synchronization of online communications, and increases revenue for online communications service providers.

The processes and procedures described herein, in addition to those explicitly described, can be implemented by software, hardware, or any combination thereof. For example, the processes and procedures described herein can be implemented by implementing logic corresponding to the processes and procedures in a medium such as an integrated circuit, volatile memory, non-volatile memory, non-transitory computer-readable medium, magnetic disk, or the like. Furthermore, the processes and procedures described herein can be implemented as computer programs corresponding to the processes and procedures, and can be executed by various computers.

The system or method described in the above embodiments may be integrated into a program stored on a non-transitory computer-readable medium, such as a solid-state storage device, an optical disk storage device, or a magnetic disk storage device. Alternatively, the program may be downloaded from a server via the Internet and executed by a processor.

Although the technical contents and features of the present invention have been described above, those having ordinary knowledge in the technical field to which the present invention pertains can make many further variations and modifications without departing from the teachings and disclosure of the present invention. Therefore, the scope of the present invention is not limited to the embodiments already disclosed, but includes other variations and modifications that do not depart from the present invention, and is within the scope of the claims.

S1 Screen RA Area RB Area RC Area RD Area O1 Object O11 Special effect T1 Tool 1 System 10 User terminal 120 Processing unit set 122 Display 140 Processing unit set 142 Display 30 Back-end server 32 Message unit 40 Streaming server 50 Shape detection unit 90 Network S800, S802, S804, S806, S808, S810, S812, S814, S816, S818, S820, S822, S824, S828, S830, S832, S834, S836 Step 10A User terminal 10B User terminal

Claims (15)

displaying, on a user terminal, a call screen including a first display area including an image of a first user and a second display area including an image of a second user participating in a call with the first user;
displaying a first pattern on the call screen ;
comparing the first pattern with a portion of an image of the first user and a portion of an image of the second user displayed on the call screen ;
displaying a special effect on the call screen based on a result of the comparison;
A method comprising: The method of claim 1 , wherein the position of the portion of the image of the first user and the position of the portion of the image of the second user are adjusted in real time. 2. The method of claim 1, characterized in that comparing the first pattern with the portion of the video of the first user and the portion of the video of the second user includes calculating a similarity coefficient between the first pattern and a second pattern consisting of the portion of the video of the first user and the portion of the video of the second user . 4. The method of claim 3, further comprising the step of displaying the special effect if the similarity coefficient is greater than or equal to a predetermined value. moreover,
periodically calculating the similarity coefficient between the first pattern and the second pattern;
The method of claim 3, comprising : 6. The method of claim 5, further comprising the step of displaying a message on the user terminal indicating that synchronization is acceptable if the periodically calculated similarity coefficient is greater than or equal to a predetermined value for a predetermined period of time. 4. The method of claim 3, further comprising recognizing the second pattern in a gesture recognition process, a skin recognition process, a contour recognition process, a shape detection process, an object recognition process, an image comparison process, or a moving object detection process. The method of claim 1 , wherein the portion of the image of the first user includes an image of a body part or a non-body part of the first user. 4. The method of claim 3, wherein the calculation of the similarity coefficient between the first pattern and the second pattern comprises a correlation calculation process, a trajectory convolution process, a normalization process, or a minimum distance determination process. The method of claim 1 , further comprising swapping the first display area and the second display area on the call screen . The method of claim 1 , wherein the first pattern spans the first display area and the second display area . 10. The method of claim 1, further comprising the step of effecting sending a gift based on the result of the comparison. 13. The method of claim 12 , wherein the value of the gift corresponds to a difficulty of matching the first pattern. one or more processors, wherein the one or more processors execute machine-readable instructions to
displaying, on a user terminal , a call screen including a first display area including an image of a first user and a second display area including an image of a second user participating in a call with the first user;
displaying a first pattern on the call screen;
comparing the first pattern with a portion of an image of the first user and a portion of an image of the second user displayed on the call screen ;
displaying a special effect on the call screen based on a result of the comparison;
A system for executing the above. A non-transitory computer readable medium containing a program for image recognition, wherein the program is configured to:
displaying, on a user terminal , a call screen including a first display area including an image of a first user and a second display area including an image of a second user participating in a call with the first user;
displaying a first pattern on the call screen;
comparing the first pattern with a portion of an image of the first user and a portion of an image of the second user displayed on the call screen ;
displaying a special effect on the call screen based on a result of the comparison;
23. A computer-readable medium for causing a computer to execute a program.