CN120282032A

CN120282032A - Image processing method and electronic equipment

Info

Publication number: CN120282032A
Application number: CN202311862964.2A
Authority: CN
Inventors: 吴乐乐; 邵涛; 司晨; 杜一啸; 盛家豪
Original assignee: Honor Device Co Ltd
Current assignee: Honor Device Co Ltd
Priority date: 2023-12-29
Filing date: 2023-12-29
Publication date: 2025-07-08
Also published as: WO2025138866A1

Abstract

The present application discloses an image processing method and electronic device, wherein the method comprises: receiving an operation of switching from the first camera to the second camera during the operation of the first camera; obtaining M frames of first images collected in real time by the first camera and M frames of second images collected in real time by the second camera after receiving the operation, wherein the Nth frame of the first image and the Nth frame of the second image are collected simultaneously, M is a positive integer greater than 1, and N is a positive integer greater than or equal to 1 and less than or equal to M; performing image fusion based on the M frames of the first image and the M frames of the second image to generate an M-frame fused image, wherein the Nth frame of the fused image is obtained by fusion of the Nth frame of the first image and the Nth frame of the second image; after displaying the M-frame fused image, displaying the image based on the image collected by the second camera. By implementing the present application, when switching between multiple cameras, the overall displayed image can be visually presented with a slowly changing effect, thereby achieving a smooth transition of the image.

Description

Image processing method and electronic equipment

Technical Field

The embodiment of the application relates to the technical field of terminals, in particular to an image processing method and electronic equipment.

Background

At present, a plurality of cameras can be installed on the same terminal, and the rich shooting requirements of users are met through different focal length ranges provided by the cameras. For example, the focal length range provided by the wide-angle camera is 0.4x to 1.0x, the focal length range provided by the main camera is 1.0x to 3.5x, and if the user adjusts the shooting focal length from 0.9x to 1.2x, the terminal will display from the picture acquired based on the wide-angle camera, and switch to the picture acquired based on the main camera for display. Because the shooting parameters of the cameras are different and the installation positions are different, the angles of view obtained by the cameras when shooting the same object have deviation, and therefore, when the terminal is switched from displaying the picture acquired based on the former camera to displaying the picture acquired based on the latter camera, a jump phenomenon can appear visually.

Generally, the overall image acquired by the previous camera may be shifted through the transformation matrix to align the overall image acquired by the next camera as much as possible. However, since the transformation matrix is obtained based on the same plane in the whole picture acquired by the two cameras, when the whole picture acquired by the two cameras comprises a plurality of planes, the whole offset is performed based on the transformation matrix, so that the planes corresponding to the transformation matrix in the whole picture can be aligned, other planes cannot be aligned, and the transition phenomenon still exists visually when the planes which cannot be aligned are displayed in a switching mode. Therefore, how to achieve smooth transition on the whole screen during multi-shot switching is one of the problems to be solved.

Disclosure of Invention

The application provides an image processing method and electronic equipment, which realize smooth transition on an integral picture during multi-shot switching.

In a first aspect, an embodiment of the present application provides an image processing method, including:

The method comprises the steps of receiving operation of switching from a first camera to a second camera in the working process of the first camera, acquiring M frames of first images acquired in real time by the first camera and M frames of second images acquired in real time by the second camera after the operation is received, acquiring N frames of first images and N frames of second images at the same time, wherein M is a positive integer greater than 1, N is a positive integer greater than or equal to 1 and less than or equal to M, performing image fusion based on the M frames of first images and the M frames of second images to generate M frames of fused images, fusing the N frames of fused images by the N frames of first images and the N frames of second images, and displaying the images acquired based on the second camera after the M frames of fused images are displayed.

By implementing the method described in the first aspect, in the process of switching from display based on the picture of the first camera to display based on the picture of the second camera, multi-frame fusion images of the first camera and the second camera are displayed first, and then the picture of the second camera is displayed independently.

In one possible implementation manner, the image fusion is performed based on the M-frame first image and the M-frame second image, and the generation of the M-frame fusion image comprises obtaining a target fusion image based on a first display image corresponding to the target fusion image, a second display image corresponding to the target fusion image and a weight pair corresponding to the target fusion image, wherein the first display image corresponding to the target fusion image is obtained by performing image processing on the target first image, the second display image corresponding to the target fusion image is obtained by performing image processing on the target second image, the target fusion image is one frame image in the M-frame fusion image, the target first image is one frame image corresponding to the target fusion image in the M-frame first image, and the target second image is one frame image corresponding to the target fusion image in the M-frame second image.

It can be seen that the target fusion image can be obtained by performing image fusion based on the first display image, the second display image and the weight pair corresponding to the target fusion image.

In one possible implementation manner, the target fusion image is obtained based on the first display image corresponding to the target fusion image, the second display image corresponding to the target fusion image and the weight pair corresponding to the target fusion image, wherein the weight pair corresponding to the target fusion image is obtained when the first view angle corresponding to the first camera is greater than or equal to the second view angle corresponding to the second camera, the weight pair corresponding to the target fusion image is based on the weight pair corresponding to the target fusion image, the first display image corresponding to the target fusion image and the second display image corresponding to the target fusion image are subjected to weighted fusion, the target fusion image is obtained, the scale of the first display image is the same as that of the second display image, and the center point of the first display image and the center point of the second display image are the same.

Since the scale of the first display image is the same as the scale of the second display image and the center point of the first display image is aligned to the two center points of the second display image in the case that the first angle of view is greater than or equal to the second angle of view (i.e., zoomin), the first display image and the second display image do not need to be subjected to scale alignment and center point alignment by clipping when the first display image is obtained (because the center point alignment can be performed by clipping), and the target fusion image can be obtained directly.

In one possible implementation manner, the weight pair corresponding to the target fusion image comprises a first weight corresponding to the target fusion image and a second weight corresponding to the target fusion image, the weight pair corresponding to the target fusion image is used for carrying out weighted fusion on a first display image corresponding to the target fusion image and a second display image corresponding to the target fusion image to obtain the target fusion image, the weight pair corresponding to the target fusion image comprises the step of weighting the first display image corresponding to the target fusion image based on the first weight corresponding to the target fusion image to obtain a first weighted image, the weight pair corresponding to the target fusion image is used for weighting the second display image corresponding to the target fusion image to obtain a second weighted image, and the first weighted image and the second weighted image are fused to obtain the target fusion image.

In one possible implementation manner, the method comprises the steps of obtaining the target fusion image based on the first display image corresponding to the target fusion image, the second display image corresponding to the target fusion image and the weight pair corresponding to the target fusion image, wherein the step of downsampling the first display image corresponding to the target fusion image under the condition that the first angle of view corresponding to the first camera is smaller than the second angle of view corresponding to the second camera to obtain a thumbnail image corresponding to the target fusion image, the scale of the thumbnail image is the same as the scale of the second display image corresponding to the target fusion image, the step of obtaining a parameter to be aligned corresponding to the target fusion image, the step of aligning the thumbnail image to the center point of the second display image corresponding to the target fusion image based on the parameter to be aligned to obtain an aligned image, and the step of weighting and fusing the aligned image and the second display image based on the weight pair corresponding to the target fusion image to obtain the target fusion image.

Since in case the first angle of view is smaller than the second angle of view (i.e. in case of Zoom out), the scale of the first rendered image is not the same as the scale of the second rendered image (since the first rendered image and the second rendered image cannot be scale-aligned by clipping when the first rendered image is obtained), and the center point of the second rendered image is not completely aligned to the two center points of the first rendered image (since the second rendered image is obtained with partial alignment by center point offset), the first rendered image may be downsampled first, and then the first rendered image and the second rendered image are scale-aligned by downsampling, followed by center point alignment and weighted fusion, thereby obtaining the target fusion image.

In one possible implementation manner, the weight pair corresponding to the target fusion image comprises a first weight corresponding to the target fusion image and a second weight corresponding to the target fusion image, wherein the weight pair corresponding to the target fusion image is used for carrying out weighted fusion on the aligned image and the second display image to obtain the target fusion image, the method comprises the steps of weighting the aligned image based on the first weight corresponding to the target fusion image to obtain a first weighted image, weighting the second display image corresponding to the target fusion image based on the second weight corresponding to the target fusion image to obtain a second weighted image, and fusing the first weighted image and the second weighted image to obtain the target fusion image.

In one possible implementation manner, the obtaining the parameters to be aligned corresponding to the target fusion image includes obtaining a total alignment parameter and a completed alignment parameter corresponding to the target fusion image, wherein the total alignment parameter is used for indicating a center point difference value between the target first image and the target second image, the completed alignment parameter is used for indicating a parameter with a completed center point offset when the first display image and the second display image are obtained, the value of the completed alignment parameter is smaller than that of the total alignment parameter, and the parameters to be aligned corresponding to the target fusion image are determined based on the difference value of the total alignment parameter and the completed alignment parameter.

Therefore, based on the parameter to be aligned, the center points of the two images to be fused can be completely aligned before the images are fused.

In one possible implementation manner, the first weight corresponding to the nth frame of fusion image is greater than the first weight corresponding to the n+1th frame of fusion image, and the second weight corresponding to the nth frame of fusion image is smaller than the second weight corresponding to the n+1th frame of fusion image, the number of multiframe of fusion images is M, M is a positive integer greater than 1, and N is a positive integer greater than or equal to 1 and smaller than M.

Therefore, as the number of frames in the M-frame fusion image increases, the duty ratio of the picture collected by the first camera in the fusion image is gradually reduced, and the duty ratio of the picture collected by the second camera in the frame fusion image is gradually increased, so that reasonable transition of the display image can be realized.

In a second aspect, embodiments of the present application provide an electronic device that includes one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, the one or more memories being operable to store computer program code comprising computer instructions that, when executed by the one or more processors, cause the electronic device to perform the image processing method of the first aspect and any of the possible implementations thereof.

In a third aspect, an embodiment of the present application provides an image processing apparatus comprising a function/unit for performing the image processing method in the first aspect and any one of its possible implementation manners.

In a fourth aspect, an embodiment of the present application provides a chip system, where the chip system is applied to an electronic device, and the chip system includes at least one processor and an interface, where the interface is configured to receive computer instructions and transmit the computer instructions to the at least one processor, and the at least one processor executes the computer instructions to cause the electronic device to perform the image processing method in any one of the possible implementation manners of the first aspect and the first aspect.

In a fifth aspect, the present application provides a computer readable storage medium having stored therein computer instructions which, when run on an electronic device, cause the electronic device to perform the image processing method of the first aspect and any one of the possible implementation manners thereof.

In a sixth aspect, the present application provides a computer program product for, when run on a computer, causing the computer to perform the image processing method of the first aspect and any one of the possible implementations thereof.

It will be appreciated that the above-mentioned electronic device, image processing apparatus, chip system, computer readable storage medium, and computer program product may refer to the advantages of the first aspect and any possible implementation thereof, and will not be described here again.

Drawings

Fig. 1 is a schematic diagram of a preview interface before and after camera switching according to an embodiment of the present application;

fig. 2 is a schematic hardware structure of an electronic device according to an embodiment of the present application;

Fig. 3 is a schematic software structure of an electronic device according to an embodiment of the present application;

fig. 4 is a schematic flow chart of an image processing method according to an embodiment of the present application;

Fig. 5 is an interface schematic diagram of a camera APP according to an embodiment of the present application;

FIG. 6 is a schematic diagram of software interaction of an image processing method according to an embodiment of the present application;

FIG. 7 is an interaction schematic diagram of an IPE module and an image fusion module according to an embodiment of the present application;

Fig. 8 is a schematic structural diagram of a chip system according to an embodiment of the present application.

Detailed Description

The technical solutions of the embodiments of the present application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiment of the present application, unless otherwise indicated, "/" means or, for example, a/B may represent a or B, and "and/or" in the text is merely an association relationship describing an association object, which means that three relationships may exist, for example, a and/or B, and that three cases of a alone, a and B together, and B alone exist, and further, in the description of the embodiment of the present application, "a plurality" means two or more.

The terms "first," "second," and the like, are used below for descriptive purposes only and are not to be construed as implying or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature, and in the description of embodiments of the application, unless otherwise indicated, the meaning of "a plurality" is two or more.

The term "User Interface (UI)" in the following embodiments of the present application is a media interface for interaction and information exchange between an application program or an operating system and a user, which enables conversion between an internal form of information and a form acceptable to the user. The user interface is a source code written in a specific computer language such as java, extensible markup language (extensible markup language, XML) and the like, and the interface source code is analyzed and rendered on the electronic equipment to finally be presented as content which can be identified by a user. A commonly used presentation form of a user interface is a graphical user interface (graphic user interface, GUI), which refers to a graphically displayed user interface that is related to computer operations. It may be a time, date, text, icon, button, menu, tab, text box, dialog box, status bar, navigation bar, widget, etc. visual interface element displayed in the display of the electronic device.

In order to facilitate understanding of the solution provided by the embodiments of the present application, the following description will refer to the related terms related to the embodiments of the present application:

1. Angle of view

The field of view (FOV) refers to an angle formed by two edges of the maximum range of a lens of a camera, that is, the angle of view is the range of angles in which the camera can receive images. The field angle of the camera is proportional to the field of view of the camera. For example, the larger the field of view of the camera is when the field of view is larger, and the smaller the field of view of the camera is when the field of view is smaller. In general, when the focal length of a camera is changed, the angle of view of the camera is changed, for example, as the focal length is increased, the angle of view of the camera is reduced, the number of objects that the camera can capture is reduced and the objects that are captured are larger in a picture, whereas as the focal length is reduced, the angle of view of the camera is increased, the number of objects that the camera can capture is increased and the objects that are captured are smaller in a picture.

2. Dimension of

The number of pixel points occupied by the object shot by the camera in the picture is used for indicating the size of the picture. The larger the scale of the picture, the more blurred the picture, and the smaller the scale of the picture, the finer the picture. For example, when the focal length is increased, the photographed object is larger in the picture (corresponding to the object occupying more and more pixels), the smaller the image scale is, the finer the picture is visually, whereas when the focal length is reduced, the photographed object is smaller in the picture (corresponding to the object occupying less and more pixels), and the larger the image scale is, the more the picture is blurred visually.

3. Difference between center points

When the electronic apparatus is equipped with a plurality of cameras, the plurality of cameras are different in mounting positions, and the plurality of cameras are different in optical center. Therefore, the center point of the screen is different for the screens photographed by the plurality of cameras. The difference between the center points of the pictures photographed by the plurality of cameras is referred to as a difference between the center points.

4. Multi-shot switching and multi-shot smoothing

In the process that the electronic device shoots and displays the preview picture through the camera, if the electronic device displays the picture acquired based on the previous camera and switches to the picture acquired based on the next camera for displaying, the process is called multi-shot switching. In the application, the multi-shot switching can be triggered based on the mode of adjusting the focal length by a user. For example, the electronic device is provided with three cameras, the three cameras are a wide-angle camera, a main camera and a tele camera, wherein the focal length range of the wide-angle camera is 0.4x to 1.0x, the focal length range of the main camera is 1.0x to 3.5x, and the focal length range of the tele camera is 3.5x to 30x, and when a user adjusts the focal length to switch from the focal length range corresponding to one camera to the focal length range corresponding to the other camera, the electronic device triggers multi-shot switching.

Because the shooting parameters (such as focal length, brightness, color, etc.) of the previous camera are different from those of the next camera, and the center point of the picture of the previous camera is different from that of the picture of the next camera, the displayed picture can jump when the electronic equipment switches multiple cameras.

In order to eliminate the jump phenomenon existing in the multi-shot switching as much as possible, multi-shot smoothing is needed, wherein the traditional multi-shot smoothing mode can be realized by a transformation matrix (such as a warp matrix) and the like, wherein the transformation matrix is obtained by solving the relation between pictures before and after the multi-shot switching. The shift processing can be performed on the picture before switching by using the transformation matrix, so that the picture obtained by the shift processing is aligned to the picture after switching as much as possible. Then, by displaying the picture obtained by the offset processing first and then displaying the switched picture, the multi-shot switching smoothness can be realized. However, since the transformation matrix has homography, the picture obtained by the offset processing based on the transformation matrix cannot be completely aligned to the switched picture. For example, as shown in fig. 1, the left side in fig. 1 is the preview screen of the previous camera, and the right side in fig. 1 is the preview screen of the next camera, where both preview screens include a person, a road behind the person, and a tree far away, and the person is not on the same plane in the screen as the road and the tree. When the multi-shot smoothing is performed, a transformation matrix can be calculated according to the relationship of the characters in the two preview pictures, and then the left preview picture is aligned to the right preview picture by utilizing the transformation matrix. The transformation matrix is obtained based on the relation of the characters in the two preview pictures, so the characters of the left preview picture can be successfully aligned to the characters of the right preview picture according to the transformation matrix, however, the characters, the roads and the trees of the left preview picture cannot be aligned to the roads and the trees of the right preview picture based on the transformation matrix because the characters, the roads and the trees are not on the same plane, and further, the multi-shot smoothing mode cannot completely eliminate the visual jumping phenomenon, so that the smooth transition of the whole picture is realized.

In order to realize smooth transition of an overall picture when multi-shot switching is realized, the embodiment of the application provides an image processing method and electronic equipment. The electronic device may be a terminal device having a plurality of cameras, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart car, etc., but is not limited thereto. The embodiment of the application does not limit the type of the electronic equipment. The electronic device may be, for example, the electronic device 200.

The hardware architecture of the electronic device 200 is described below, and fig. 2 shows a schematic diagram of the hardware architecture of the electronic device 200. It should be understood that the electronic device 200 may have more or fewer components than shown, may combine two or more components, or may have a different configuration of components. The various components shown in the figures may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.

As shown in fig. 2, the electronic device 200 may include a processor 110, a memory 120, a camera 193, a display 194, a mobile communication module 150, and a wireless communication module 160. Wherein:

The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (IMAGE SIGNAL processor, ISP), a controller, a memory, a video codec, a digital signal processor (DIGITAL SIGNAL processor, DSP), a baseband processor, and/or a neural network processor (neural-network processing unit, NPU), etc. Wherein the different processing units may be separate devices or may be integrated in one or more processors.

The controller may be a neural hub and a command center of the electronic device 200, among others. The controller can generate operation control signals according to the instruction operation codes and the time sequence signals to finish the control of instruction fetching and instruction execution.

A memory 120 may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to reuse the instruction or data, it can be called directly from the memory. Repeated accesses are avoided and the latency of the processor 110 is reduced, thereby improving the efficiency of the system.

In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an integrated circuit (inter-INTEGRATED CIRCUIT, I2C) interface, an integrated circuit built-in audio (inter-INTEGRATED CIRCUIT SOUND, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous receiver transmitter (universal asynchronous receiver/transmitter, UART) interface, a mobile industry processor interface (mobile industry processor interface, MIPI), a general-purpose input/output (GPIO) interface, a subscriber identity module (subscriber identity module, SIM) interface, and/or a universal serial bus (universal serial bus, USB) interface, among others.

The camera 193 is used to capture images or video. The object generates an optical image through the lens and projects the optical image onto the photosensitive element. The photosensitive element, or image sensor, may be a charge coupled device (charge coupled device, CCD) or a Complementary Metal Oxide Semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal and transmits the converted electrical signal to the processor 110 so that the processor 110 can process the electrical signal. In an embodiment of the present application, the electronic device 200 may include a plurality of cameras 193, where the plurality of cameras 193 are located differently.

The display screen 194 is used to display images, videos, and the like. The display 194 includes a display panel. The display panel may employ a Liquid Crystal Display (LCD) CRYSTAL DISPLAY, an organic light-emitting diode (OLED), an active-matrix organic LIGHT EMITTING diode (AMOLED), a flexible light-emitting diode (FLED), miniled, microLed, micro-oLed, a quantum dot LIGHT EMITTING diode (QLED), or the like. In some embodiments, the electronic device 200 may include one or more display screens 194.

The antennas 1 and 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the terminal device may be used to cover a single or multiple communication bands. Different antennas may also be multiplexed to improve the utilization of the antennas. For example, the antenna 1 may be multiplexed into a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 may provide a solution for wireless communication including 2G/3G/4G/5G or the like applied on a terminal device. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA), etc. The mobile communication module 150 may receive electromagnetic waves from the antenna 1, perform processes such as filtering, amplifying, and the like on the received electromagnetic waves, and transmit the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 150 can amplify the signal modulated by the modem processor, and convert the signal into electromagnetic waves through the antenna 1 to radiate. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be disposed in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be provided in the same device as at least some of the modules of the processor 110.

The modem processor may include a modulator and a demodulator. The modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used for demodulating the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low frequency baseband signal to the baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and then transferred to the application processor. The application processor outputs sound signals through an audio device or displays images or video. In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be provided in the same device as the mobile communication module 150 or other functional module, independent of the processor 110.

The wireless communication module 160 may provide solutions for wireless communication including wireless local area network (wireless local area networks, WLAN) (e.g., wi-Fi network), bluetooth (BT), BLE broadcast, global navigation satellite system (global navigation SATELLITE SYSTEM, GNSS), frequency modulation (frequency modulation, FM), near field communication (NEAR FIELD communication, NFC), infrared (IR), etc. applied on the terminal device. The wireless communication module 160 may be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the electromagnetic wave signals, filters the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 may also receive a signal to be transmitted from the processor 110, frequency modulate it, amplify it, and convert it to electromagnetic waves for radiation via the antenna 2.

In some embodiments, the antenna 1 of the terminal device is coupled to the mobile communication module 150 and the antenna 2 is coupled to the wireless communication module 160 so that the terminal device can communicate with the network and other devices through wireless communication technology. The wireless communication techniques can include the Global System for Mobile communications (global system for mobile communications, GSM), general packet radio service (GENERAL PACKET radio service, GPRS), code division multiple access (code division multiple access, CDMA), wideband code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC, FM, and/or IR techniques, among others. The GNSS may include a global satellite positioning system (global positioning system, GPS), a global navigation satellite system (global navigation SATELLITE SYSTEM, GLONASS), a beidou satellite navigation system (beidou navigation SATELLITE SYSTEM, BDS), a quasi zenith satellite system (quasi-zenith SATELLITE SYSTEM, QZSS) and/or a satellite based augmentation system (SATELLITE BASED AUGMENTATION SYSTEMS, SBAS).

The following describes the software architecture of the electronic device 200, as shown in fig. 3, and fig. 3 is a software architecture diagram of the electronic device 200 according to an embodiment of the present application. The software structure adopts a layered architecture, the layered architecture divides the software into a plurality of layers, and each layer has clear roles and division work. The layers communicate with each other through a software interface. As shown in fig. 3, taking an Android (Android) system, where the Android system operates on an AP, in some embodiments, the Android system is divided into four layers, namely, an application layer, an application framework layer (framework), a hardware abstraction layer (hardware abstraction layer, HAL), and a kernel layer (kernel) from top to bottom. Wherein:

① . Application layer the application layer may comprise a series of application packages. As shown in fig. 3, the application package includes an application program such as a camera application. For example, the camera application may be a camera APP of the system, a camera APP installed by the user, or a three-party application capable of calling the camera APP to take a photograph. In the present application, the camera application may trigger multi-shot switching and multi-shot smoothing in response to a user operation.

② . Application framework layer the application framework layer provides an application programming interface (application programming interface, API) framework and various services and management tools for application developers to access core functions, the application framework layer including a number of predefined functions and algorithms. As shown in fig. 3, the application framework layer may include a camera service, which may be invoked by a camera application in the application layer after the camera application acquires a user's operation, so that the camera service may implement multi-shot switching and multi-shot smoothing by managing states (e.g., camera on/off states), sessions, and collected data, etc. of a plurality of cameras. For example, in the case of multi-shot switching, the camera service needs to acquire related data acquired by a previous camera and a next camera, and in particular, the camera service may acquire multi-frame image data acquired in real time by the previous camera after receiving an operation of a user through a session corresponding to the previous camera. Meanwhile, the camera service needs to determine that the latter camera is in a working state, and obtains multi-frame image data acquired by the latter camera in real time after receiving the operation of the user through the session corresponding to the latter camera. Optionally, the application framework layer may further include a window manager, a content provider, a view system, a resource manager, a notification manager, and the like, and the camera application may call the content provider, the resource manager, the notification manager, the window manager, the view system, and the like to assist in managing the camera service, which is not limited in any way by the embodiment of the present application.

③ . Hardware abstraction layer-a hardware abstraction layer is an interface layer between the operating system kernel and the hardware circuitry that aims to abstract the hardware. It hides the hardware interface details of the specific platform and can provide a virtual hardware platform for the operating system. As shown in fig. 3, the hardware abstraction layer includes image hardware abstraction such as image hardware abstraction, and includes an image signal processing front end (IMAGE SINGNAL processing front end, IFE) module, a Space Alignment Transformation (SAT) module, an image signal processing back end (IMAGE SINGNAL processing post end, IPE) module, an image fusion module, and a sensor node. After receiving the image data collected by different cameras at the same time, the hardware abstraction layer sequentially processes the image data through the IFE module, the SAT module and the IPE module.

After receiving the image data collected by each camera (Sensor), the IFE module performs preliminary cropping on an initial image corresponding to the collected image data, so that the size of the image obtained after the preliminary cropping can be suitable for the size of the screen display. Optionally, the IFE module may perform color correction, demosaicing, etc. pre-processing on the initial image during the preliminary cropping.

The SAT module is used for receiving the images processed by the IFE module by the different cameras, then the SAT module calculates transformation information among the different cameras according to the plurality of images, the transformation information comprises one or more of center point alignment parameters and a cutting frame, and the transformation information is sent to the IPE module. For example, taking a third frame as an example, the IFE module receives the third frame image data acquired by the previous camera in real time, and obtains a clipping image after preliminary clipping and sends the clipping image to the SAT module. The SAT module may calculate transformation information based on the two cropped images corresponding to the obtained third frame.

The IPE module is configured to shift (i.e. center point alignment) the overall frames corresponding to the cropping images of the different cameras according to the transformation information calculated by the SAT module, and/or adjust the angles of view corresponding to the cropping images of the different cameras to the same size (i.e. scale alignment) through the cropping frame. Optionally, the IPE module may also perform noise reduction, detail enhancement, color processing, etc. on the cut images of different cameras.

The image fusion module is used for receiving the image obtained after the IPE module is processed and carrying out image fusion on the received image. Optionally, if the images obtained after processing by the IPE module are not spatially aligned (including scale alignment and center point alignment), the image fusion module needs to spatially align again before image fusion, and then performs image fusion. The sensor node is used for receiving an instruction of a camera service in the application framework layer to control the on or off of the camera.

④ . The kernel layer is a layer between hardware and software. The kernel layer comprises camera drivers, display drivers and the like, and each driver is used for responding to the instruction of the corresponding hardware abstraction in the hardware abstraction layer so as to realize the corresponding function. For example, the camera driver may receive instructions from a sensor node in the hardware abstraction layer to turn on or off the camera, and the display driver may receive instructions from an image fusion module in the hardware abstraction layer to display a multi-frame fused image.

It can be appreciated that the above software architecture is merely an example, and in a specific implementation, the electronic device 200 may further include further functional modules in the above layers, which is not repeated in the present application.

The image processing method provided by the embodiment of the application is further described in detail below:

as shown in fig. 4, fig. 4 is a flowchart of an image processing method according to an embodiment of the present application. The method execution subject shown in fig. 4 may be an electronic device, or the subject may be a chip in the electronic device. Fig. 4 illustrates an example of an execution body of the method of the electronic device. The main principles of the method shown in other drawings of the embodiment of the present application are the same, and the software and hardware structures of the electronic device may refer to the structures shown in fig. 2 and fig. 3. The image processing method shown in fig. 4 includes steps 401 to 404. Wherein:

401. in the working process of the first camera, the electronic equipment receives the operation of switching from the first camera to the second camera.

In the embodiment of the application, the electronic equipment is provided with a plurality of cameras, and the shooting parameters, the installation positions and the like of the cameras are different. The first camera is any one of a plurality of cameras, and the second camera is any one of a plurality of cameras except the first camera. For example, if the electronic device is configured with a main camera, a wide-angle camera, and a telephoto camera, the first camera may be the main camera, and the second camera may be the wide-angle camera.

The first camera work means that the electronic equipment shoots through the first camera and continuously displays a preview picture when the first camera shoots. In the working process of the first camera, the electronic equipment can receive the operation of switching the first camera to the second camera from the user. For example, when the focal length range supported by the first camera is different from the focal length range supported by the second camera, the operation may be an operation of adjusting the user from the first focal length to the second focal length, the first focal length being a focal length in the focal length range supported by the first camera, the second focal length being a focal length in the focal length range supported by the second camera. For example, fig. 5 shows an interface schematic diagram of a camera APP, where an interface 501 is an interface displayed by an electronic device when a main camera is used for shooting, the interface 501 includes a preview interface and a shooting functional area, the shooting functional area includes multiple shooting modes, an album control, a shooting control and a front-back switching control, and the multiple shooting modes include a "night scene" mode, a "portrait" mode, a "large aperture" mode, a "shooting" mode, a "video" mode, a "professional" mode and a "more" mode. In the preview interface, a photographing object and a focal length setting area are displayed, and the focal length setting area in the interface 501 indicates that the photographing focal length of the current main camera is 1x. After the focal length setting area receives the operation of sliding upwards by the user, as shown in the interface 502, the electronic device increases the focal length of the camera, and determines whether the target focal length expected by the user is still the focal length that the main camera can support, for example, the target focal length is 4x, the focal length range of the main camera is 1x-3.5x, and the focal length range of the tele camera is 3.5 x-30 x, which indicates that the target focal length is not in the focal length range of the main camera but in the focal length range of the tele camera, so that the electronic device displays the interface shown in the interface 503, the interface 503 is a shooting preview interface corresponding to the tele camera, and the size of the shooting object in the interface 503 becomes large. It can be understood that, after the focal length setting area receives the downward sliding operation of the user, the electronic device reduces the focal length of the camera, determines whether the target focal length expected by the user is still the focal length that can be supported by the main camera, and when the target focal length expected by the user is not the focal length that can be supported by the main camera, triggers the switch from the main camera to the rest of cameras, and displays the shooting preview interfaces corresponding to the rest of cameras.

Optionally, after the electronic device receives the operation of switching from the first camera to the second camera, the state of the second camera may be determined first, if the second camera is in the start state, the second camera is maintained in the start state, if the second camera is in the close state, the second camera is started, and the second camera is updated from the close state to the start state. This is because image signal processing in the electronic device can receive image data acquired from two cameras at most simultaneously, so the electronic device can support the simultaneous starting of the two cameras at most. If the second camera is not started before the operation is received, the electronic device starts the second camera, and if the second camera is started before the operation is received, the electronic device does not need to start the second camera. It should be noted that, after the operation is received and before the second camera is successfully started, the electronic device may continuously display the preview image captured by the first camera.

402. The electronic equipment acquires M frames of first images acquired in real time by the first camera and M frames of second images acquired in real time by the second camera after receiving the operation.

After the electronic device receives the operation, the electronic device can acquire multi-frame images in real time through the first camera and the second camera. The application refers to an image acquired by a first camera in real time as a first image, wherein the first image is an initial image which is not subjected to image processing, and an image acquired by a second camera in real time as a second image, wherein the second image is an initial image which is not subjected to image processing.

Assuming that the first camera collects M frames of first images and the second camera collects M frames of second images, M is a positive integer greater than 1, then an nth frame of first images in the M frames of first images and an nth frame of second images in the M frames of second images are collected simultaneously, and N is a positive integer greater than or equal to 1 and less than or equal to M.

After acquiring the M-frame first image and the M-frame second image, the electronic device may perform the subsequent step 403 to obtain and display the M-frame fused image, and then perform the subsequent step 404. Therefore, the electronic equipment can display the images obtained by fusing the plurality of first cameras and the plurality of second cameras, and then display the images based on the images acquired by the second cameras independently, so that the switching transition of the cameras is realized.

403. And the electronic equipment performs image fusion based on the M-frame first image and the M-frame second image to generate an M-frame fusion image.

In the embodiment of the present application, the value of M may be preset. The electronic equipment can generate any frame fusion image in the M frame fusion images in real time, namely, after the electronic equipment acquires the first image of the N frame and the second image of the N frame in real time, the electronic equipment can perform image fusion based on the first image of the N frame and the second image of the N frame to acquire the fusion image of the N frame, then, the electronic equipment can acquire the first image of the N+1 frame and the second image of the N+1 frame in real time, and perform image fusion based on the first image of the N+1 frame and the second image of the N+1 frame to acquire the fusion image of the N+1 frame, and the electronic equipment repeats the operations until the fusion image of the M frame is acquired.

In one possible implementation manner, the target fusion image is one frame of image in the M frames of fusion images, and the electronic device may obtain the target fusion image based on the first display image corresponding to the target fusion image, the second display image corresponding to the target fusion image, and the weight pair corresponding to the target fusion image.

The target first image is one frame of image corresponding to the target fusion image in the M frames of first images, and the target second image is one frame of image corresponding to the target fusion image in the M frames of second images. For example, the target fusion image is a third frame fusion image, and the target first image is a third frame first image, and the target second image is a third frame second image.

The first display image corresponding to the target fusion image is obtained by performing image processing on the target first image, and the second display image corresponding to the target fusion image is obtained by performing image processing on the target second image.

Specifically, after the electronic device obtains the target first image and the target second image, the first target image and the second target image may be initially cropped by the IFE module, so that the sizes of the two images obtained by initial cropping are adapted to the size of the screen display. Then, the electronic equipment performs image processing on the two images obtained by initial cutting through the SAT module and the IPE module according to the frame number corresponding to the target fusion image to obtain a first display image corresponding to the target fusion image and a second display image corresponding to the target fusion image. And then according to the frame number corresponding to the target fusion image, fusing the first display image corresponding to the target fusion image and the second display image corresponding to the target fusion image through an image fusion module and a weight pair corresponding to the target fusion image to obtain the target fusion image.

The following describes how to obtain a first display image corresponding to the target fusion image and a second display image corresponding to the target fusion image:

Assuming that the target fusion image is an nth frame fusion image in the M frame fusion images, N is a positive integer greater than or equal to M and less than M, the target first image is an nth frame first image in the M frame first images, and the target second image is an nth frame second image in the M frame second images.

Firstly, the electronic equipment respectively performs initial cutting on an N-th frame first image and an N-th frame second image through an IFE module, and calculates two images obtained through initial cutting through a SAT module to determine a total alignment parameter and a first alignment parameter corresponding to an N-th frame fusion image. And then, the electronic equipment shifts the two images obtained by initial cutting based on the first alignment parameters corresponding to the N-th frame fusion image through the IPE module to obtain a first display image corresponding to the N-th frame fusion image and a second display image corresponding to the N-th frame fusion image.

The total alignment parameter is used for indicating a center point difference value (corresponding to a center point difference value between an N-th frame first image and an N-th frame second image) of the two images obtained by initial cutting, and the first alignment parameter corresponding to the N-th frame fusion image is used for indicating a parameter to be aligned by the IPE module.

In the conventional multi-shot switching transition process, under Zoomin scenes, the electronic device can determine that the first alignment parameter is the same as the total alignment parameter, so that when the electronic device shifts two images obtained by initial cutting through the IPE module, the center point of the first sending image corresponding to the obtained N-th frame fusion image can be completely aligned to the center point of the second sending image corresponding to the N-th frame fusion image. However, in the Zoom out scene, the electronic device does not directly set the first alignment parameter to be the same as the total alignment parameter, and the electronic device gradually increases the value of the first alignment parameter along with the increase of the value of N, so that when the electronic device shifts two images obtained by initial cropping through the IPE module, the center point of the first display image corresponding to the nth frame fusion image cannot be completely aligned to the center point of the second display image corresponding to the nth frame fusion image.

For example, if the first angle of view corresponding to the first camera is greater than or equal to the second angle of view corresponding to the second camera, the scene indicating the multi-shot switching is Zoomin. Assuming that the value of M is 5, and the difference between the center points of the first image and the second image is 10 pixels (i.e., the total alignment parameter is 10 pixels), the first alignment parameter corresponding to the first frame fusion image, the first alignment parameter corresponding to the second frame fusion image, the first alignment parameter corresponding to the third frame fusion image, the first alignment parameter corresponding to the fourth frame fusion image, and the first alignment parameter corresponding to the fifth frame fusion image are all 10 pixels. Therefore, the electronic device can enable the center point of the first display image corresponding to the first frame fusion image to be completely aligned to the center point of the second display image corresponding to the first frame fusion image through the IPE module, the center point of the first display image corresponding to the second frame fusion image to be completely aligned to the center point of the second display image corresponding to the second frame fusion image, the center point of the first display image corresponding to the third frame fusion image to be completely aligned to the center point of the second display image corresponding to the third frame fusion image, the center point of the first display image corresponding to the fourth frame fusion image to be completely aligned to the center point of the second display image corresponding to the fourth frame fusion image, and the center point of the first display image corresponding to the fifth frame fusion image to be completely aligned to the center point of the second display image corresponding to the fifth frame fusion image.

For another example, if the first field angle corresponding to the first camera is smaller than the second field angle corresponding to the second camera, the scene indicating the multi-shot switching is a Zoom out scene. Assuming that the value of M is 5 and the difference between the center points of the first image and the second image is 10 pixels (i.e., the total alignment parameter is 10 pixels), the first alignment parameter corresponding to the first frame fusion image, the first alignment parameter corresponding to the second frame fusion image, the first alignment parameter corresponding to the third frame fusion image, the first alignment parameter corresponding to the fourth frame fusion image, and the first alignment parameter corresponding to the fifth frame fusion image may be 1 pixel, 2 pixels, 4 pixels, 6 pixels, and 8 pixels, respectively. Because the first alignment parameter is smaller than the total alignment parameter, the electronic device through the IPE module cannot completely align the center point of the first display image corresponding to the first frame fusion image to the center point of the second display image corresponding to the first frame fusion image, cannot completely align the center point of the first display image corresponding to the second frame fusion image to the center point of the second display image corresponding to the second frame fusion image, cannot completely align the center point of the first display image corresponding to the third frame fusion image to the center point of the second display image corresponding to the third frame fusion image, cannot completely align the center point of the first display image corresponding to the fourth frame fusion image to the center point of the second display image corresponding to the fourth frame fusion image, and cannot completely align the center point of the first display image corresponding to the fifth frame fusion image to the center point of the second display image corresponding to the fifth frame fusion image.

Optionally, if the first field angle corresponding to the first camera is greater than the second field angle corresponding to the second camera, the electronic device may further obtain, through the SAT module, re-cropping information when shifting based on the first alignment parameter, and re-cropping the two images obtained by the initial cropping according to the re-cropping information through the IPE module.

For example, the first view angle is larger than the second view angle, and the field of view of the nth frame of initial cropping image (i.e., the image obtained by performing initial cropping on the nth frame of first image by the IFE module) corresponding to the first camera is larger than the field of view of the nth frame of initial cropping image (i.e., the image obtained by performing initial cropping on the nth frame of second image by the IFE module) corresponding to the second camera. Therefore, the re-clipping information of the N-th frame initial clipping image corresponding to the first camera can be obtained, the image with the same content as the N-th frame initial clipping image corresponding to the second camera is clipped from the N-th frame initial clipping image corresponding to the first camera through the IPE module, and the clipped image is amplified, so that the scale of the first display image corresponding to the N-th frame fusion image is the same as the scale of the second display image corresponding to the N-th frame fusion image.

If the first angle of view is smaller than the second angle of view, the field of view of the nth frame of initial cropping image (i.e., the image obtained by the IFE module initially cropping the nth frame of first image) corresponding to the first camera is smaller than the field of view of the nth frame of initial cropping image (i.e., the image obtained by the IFE module initially cropping the nth frame of second image) corresponding to the second camera. Therefore, the content of the nth frame initial cropping image corresponding to the first camera is a part of the content of the nth frame initial cropping image corresponding to the second camera, and the nth frame initial cropping image corresponding to the first camera cannot be cropped again to obtain an image with the same content as the nth frame initial cropping image corresponding to the second camera. Therefore, under the condition that the first view angle is smaller than the second view angle, the electronic device does not need to obtain the re-cropping information through the SAT module, and the IPE module does not need to re-crop the two images obtained through initial cropping according to the re-cropping information. If the first view angle is equal to the second view angle, the electronic equipment does not need to be cut again, namely the electronic equipment does not need to obtain cutting information again through the SAT module, and the IPE module does not need to cut again.

After the first display image corresponding to the target fusion image and the second display image corresponding to the target fusion image are obtained based on the operations, the target fusion image can be obtained based on the first display image corresponding to the target fusion image and the second display image corresponding to the target fusion image.

How to obtain the target fusion image based on the first display image corresponding to the target fusion image and the second display image corresponding to the target fusion image is described below:

(1) In a first possible implementation manner, under the condition that the first view angle is greater than or equal to the second view angle, the electronic device acquires a weight pair corresponding to the target fusion image, and performs weighted fusion on a first display image corresponding to the target fusion image and a second display image corresponding to the target fusion image based on the weight pair corresponding to the target fusion image to obtain the target fusion image.

Under the condition that the first view angle is larger than or equal to the second view angle, according to the operation of the SAT module and the IPE module, the scale of the first display image corresponding to the target fusion image is the same as that of the second display image corresponding to the target fusion image, and the center point of the first display image corresponding to the target fusion image is completely aligned to the center point of the second display image corresponding to the target fusion image. Therefore, the electronic device can perform weighted fusion on the first display image corresponding to the target fusion image and the second display image corresponding to the target fusion image based on the weight pair corresponding to the target fusion image, so as to obtain the target fusion image.

Specifically, the weight pair corresponding to the target fusion image includes a first weight corresponding to the target fusion image and a second weight corresponding to the target fusion image. The electronic equipment firstly weights a first display image corresponding to the target fusion image based on a first weight corresponding to the target fusion image to obtain a first weighted image, weights a second display image corresponding to the target fusion image based on the first weight corresponding to the target fusion image to obtain a second weighted image, and then fuses the first weighted image and the second weighted image to obtain the target fusion image.

(2) In a second possible implementation manner, under the condition that a first field angle corresponding to a first camera is smaller than a second field angle corresponding to a second camera, the electronic device downsamples a first display image corresponding to the target fusion image to obtain a thumbnail image corresponding to the target fusion image, acquires parameters to be aligned corresponding to the target fusion image, aligns the thumbnail image to a center point of the second display image based on the parameters to be aligned to obtain an aligned image, and performs weighted fusion on the aligned image and the second display image based on a weight pair corresponding to the target fusion image to obtain the target fusion image.

Under the condition that the first view angle is smaller than the second view angle, according to the operations of the SAT module and the IPE module, the scale of the first display image corresponding to the target fusion image is different from the scale of the second display image corresponding to the target fusion image, and the center point of the second display image corresponding to the target fusion image is not completely aligned to the center point of the first display image corresponding to the target fusion image. Therefore, the electronic device can perform scale alignment and center point alignment on the first display image corresponding to the target fusion image and the second display image corresponding to the target fusion image, and then perform weighted fusion on the images after scale alignment and center point alignment to obtain the target fusion image.

Specifically, the electronic device downsamples a first display image corresponding to the target fusion image to obtain a thumbnail image, so that the scale of the thumbnail image is the same as the scale of a second display image corresponding to the target fusion image, and scale alignment is achieved.

Then, the electronic device determines a parameter to be aligned corresponding to the target fusion image based on a difference value between the total alignment parameter and a completed alignment parameter corresponding to the target fusion image (the completed alignment parameter is a first alignment parameter corresponding to the target fusion image in the above content), for example, the total alignment parameter is 10 pixels, the first alignment parameter corresponding to the first frame fusion image is 1 pixel, and the parameter to be aligned corresponding to the first frame image is 9 parameters. According to the parameters to be aligned corresponding to the target fusion image, the electronic equipment aligns the center point of the thumbnail image to the center point of the second display image corresponding to the target fusion image, and an aligned image is obtained, so that center point alignment is realized.

Finally, the weight pair corresponding to the target fusion image comprises a first weight corresponding to the target fusion image and a second weight corresponding to the target fusion image, the electronic equipment can weight the alignment image based on the first weight corresponding to the target fusion image to obtain a first weighted image, and weight the second display image corresponding to the target fusion image based on the first weight corresponding to the target fusion image to obtain a second weighted image, and the first weighted image and the second weighted image are fused to obtain the target fusion image.

Optionally, the electronic device may adjust the first weight and the second weight corresponding to the target fusion weights related in (1) and (2) according to the number of frames corresponding to the target fusion image. For example, the first weight corresponding to the nth frame fused image is greater than the first weight corresponding to the n+1th frame fused image, and the second weight corresponding to the nth frame fused image is less than the second weight corresponding to the n+1th frame fused image.

For example, the first weight may gradually decrease from 1 to 0 as the number of frames increases, and the second weight may gradually decrease from 0 to 1 as the number of frames increases. Because the first weight is used for weighting the aligned image or the first display image, the aligned image and the first display image are both derived from the picture acquired by the first camera, the second weight is used for weighting the second display image, and the second display image is derived from the picture acquired by the second camera. Therefore, when the first weight gradually decreases with the number of frames corresponding to the target fusion image and the second weight gradually increases, the duty ratio of the picture collected by the first camera in the target fusion image gradually decreases, and the duty ratio of the picture collected by the second camera in the target fusion image gradually increases.

404. And the electronic equipment displays the M frames of fused images based on the images acquired by the second camera.

The electronic device can display the M frame fusion image in real time, namely the electronic device displays the N frame fusion image after obtaining the N frame fusion image, then the electronic device displays the N+1st frame fusion image after obtaining the N+1st frame fusion image, and the electronic device repeats the operation until the M frame fusion image is displayed, and the electronic device can display the picture acquired by the second camera independently.

Optionally, if the first weight gradually decreases along with the frame number and the second weight gradually increases along with the frame number, the user can feel that the picture collected by the first camera gradually fades and the picture collected by the second camera gradually becomes clear when the electronic device sequentially displays the M-frame fusion images.

Based on the embodiment described in fig. 4, the electronic device may display the multi-frame fused image of the first camera and the second camera first during multi-shot switching, and then display the picture of the second camera separately.

In the following, in conjunction with the software architecture diagram of fig. 3, taking the nth frame as an example, the software interaction for obtaining the nth frame fusion image is described in detail, and the software interaction in fig. 6 only describes the interaction flow of each module in the hardware abstraction layer, and the interactions between the rest layers are not described herein.

And 601, acquiring an nth frame first image and an nth frame second image by an IFE module.

The IFE module can acquire an N frame first image acquired by the first camera and an N frame second image acquired by the second camera.

Step 602, the IFE module initially clips the first image of the nth frame and the second image of the nth frame to obtain the first clipping image of the nth frame and the second clipping image of the nth frame.

The IFE module in the electronic equipment supports the simultaneous processing of two paths of data, so that the IFE module can perform initial cropping on an N-th frame of second image while performing initial cropping on an N-th frame of first image, and respectively obtain an N-th frame of first cropping image and an N-th frame of second cropping image. The initial cropping is used to adapt the size of the resulting nth frame first cropped image and the size of the nth frame second cropped image to the size of the screen display preview picture.

Step 603, the IFE module sends the nth frame of the first clipping image and the nth frame of the second clipping image to the SAT module.

Step 604, the SAT module determines a total alignment parameter and a first alignment parameter corresponding to the nth frame of the fused image.

The SAT module can firstly determine the coordinates of the central point of the first clipping image of the N frame and the coordinates of the central point of the second clipping image of the N frame, and determine the total alignment parameter according to the difference between the two coordinates of the central points. And then, the SAT module determines a first alignment parameter corresponding to the N frame fusion image according to the first field angle corresponding to the first camera and the second field angle corresponding to the second camera.

Optionally, when the first field angle is greater than or equal to the second field angle, the SAT module determines that the first alignment parameter corresponding to the nth frame of fused image is equal to the total alignment parameter.

Optionally, when the first angle of view is smaller than the second angle of view, the SAT module determines a first alignment parameter corresponding to the nth frame of fusion image according to the number of frames corresponding to the nth frame of fusion image. For example, the SAT module may determine the first alignment parameter corresponding to the nth frame fusion image from the total alignment parameters according to a proportion of the number of frames corresponding to the nth frame fusion image to the total number of frames of the fusion image. For example, if the total frame number of the fused image is M, a value smaller than the total alignment parameter may be determined from the total alignment parameters according to the result of N/M. Optionally, as the value of N increases, the value of the first alignment parameter corresponding to the nth frame of fusion image may gradually increase.

Optionally, when the first view angle is greater than the second view angle, the SAT module further needs to determine re-cropping information corresponding to the first cropping image of the nth frame, where the re-cropping information is used for re-cropping the first cropping image of the nth frame.

Step 605, the SAT module sends the first cropping image of the nth frame, the second cropping image of the nth frame, the total alignment parameter, and the first alignment parameter corresponding to the fusion image of the nth frame to the IPE module.

Optionally, if the SAT module further determines the re-cropping information corresponding to the nth frame of the first cropping image, the SAT module further sends the re-cropping information corresponding to the nth frame of the first cropping image to the IPE module.

Step 606, the IPE module shifts the center points of the first clipping image of the nth frame and the second clipping image of the nth frame based on the first alignment parameter corresponding to the nth frame fusion image, so as to obtain a first display image corresponding to the nth frame fusion image and a second display image corresponding to the nth frame fusion image.

When the first clipping image of the nth frame and the second clipping image of the nth frame are offset based on the first alignment parameters corresponding to the fusion image of the nth frame, the IPE module can operate the image with larger view angle in the first clipping image of the nth frame and the second clipping image of the nth frame. For example, if the first angle of view is smaller than the second angle of view, the center point of the second cropping image of the nth frame is shifted, the center point of the first cropping image of the nth frame remains unchanged, and if the first angle of view is greater than or equal to the second angle of view, the center point of the first cropping image of the nth frame is shifted, and the center point of the second cropping image of the nth frame remains unchanged.

Optionally, the IPE module further performs clipping on the first clipping image of the nth frame based on the clipping information corresponding to the first clipping image of the nth frame.

In step 607, the IPE module sends the first display image corresponding to the nth frame of fusion image, the second display image corresponding to the nth frame of fusion image, the total alignment parameter and the first alignment parameter corresponding to the nth frame of fusion image to the image fusion module.

In step 608, the image fusion module determines a parameter to be aligned corresponding to the nth frame fusion image based on the first alignment parameter and the total alignment parameter corresponding to the nth frame fusion image.

The image fusion module may determine a difference between the total alignment parameter and a first alignment parameter corresponding to the nth frame of fusion image as a parameter to be aligned corresponding to the nth frame of fusion image.

And step 609-1, when the parameter to be aligned is 0, the image fusion module performs weighted fusion on the first display image corresponding to the nth frame fusion image and the second display image corresponding to the nth frame fusion image to obtain the nth frame fusion image.

When the first view angle is larger than or equal to the second view angle, the first alignment parameter corresponding to the nth frame of fusion image is equal to the total alignment parameter, so that the parameter to be aligned corresponding to the nth frame of fusion image is 0, and meanwhile, the scale of the first display image corresponding to the nth frame of fusion image is the same as the scale of the second display image corresponding to the nth frame of fusion image. Therefore, the image fusion module directly carries out weighted fusion to obtain an N frame fusion image.

And step 609-2, when the parameter to be aligned is not 0, the image fusion module performs scale alignment and center point alignment on the first display image corresponding to the nth frame fusion image and the second display image corresponding to the nth frame fusion image, and then performs weighted fusion to obtain the nth frame fusion image.

When the first view angle is smaller than the second view angle, the first alignment parameter corresponding to the nth frame of fusion image is smaller than the total alignment parameter, so that the parameter to be aligned corresponding to the nth frame of fusion image is not 0. And the scale of the first display image corresponding to the N frame fusion image is different from the scale of the second display image corresponding to the N frame fusion image, so that the image fusion module performs scale alignment and center point alignment at first and then performs weighted fusion to obtain the N frame fusion image. The corresponding implementation of step 609-2 may refer to the flow shown in fig. 7.

In the weighted fusion of step 609-1 and step 609-2, the image fusion module may obtain a weight pair corresponding to the nth frame of fused image, where the weight pair corresponding to the nth frame of fused image includes a first weight and a second weight corresponding to the nth frame of fused image. And then carrying out weighted fusion based on the first weight and the second weight to obtain an N frame fusion image.

Optionally, the image fusion module may determine a first weight and a second weight corresponding to the nth frame fusion image according to a proportion of a frame number corresponding to the nth frame fusion image to a total frame number of the fusion image, where the first weight decreases with an increase in the proportion, and the second weight increases with an increase in the proportion.

For example, if the frame number corresponding to the nth frame of the fused image is N and the total frame number of the fused image is M, the first weight and the second weight corresponding to the nth frame of the fused image may be determined according to the result of N/M. For example, the value of N/M may be directly determined as the second weight, the value of 1-N/M may be determined as the first weight, if the value of M is 5, the first weight corresponding to the first frame fusion image is 0.8, the second weight is 0.2, the first weight corresponding to the first frame fusion image is 0.6, and the second weight is 0.4. Or the first weight and the second weight which are corresponding to the N values and are different can be stored in advance, so that the image fusion module can directly read the first weight and the second weight which are corresponding to the N frame fusion image according to the N values.

Fig. 7 shows the interaction flow of the IPE module and the image fusion module when the first angle of view is smaller than the second angle of view:

As shown in fig. 7, when the first view angle corresponding to the first camera is smaller than the second view angle corresponding to the second camera, the view range of the first cropping image of the nth frame is smaller than the view range of the second cropping image of the nth frame, and the photographed object in the first cropping image of the nth frame is larger than the photographed object in the second cropping image of the nth frame, so when the IPE module shifts the center points of the first cropping image of the nth frame and the second cropping image of the nth frame, the center point of the first cropping image of the nth frame can be kept unchanged, a first display image corresponding to the nth frame fusion image is obtained, and the center point of the second cropping image of the nth frame is shifted by adopting the first alignment parameter corresponding to the nth frame fusion image, so that a second display image corresponding to the nth frame fusion image is obtained. As shown in fig. 7, since the scale of the first display image corresponding to the nth frame of fusion image is different from the scale of the second display image corresponding to the nth frame of fusion image, the size of the shooting object in the first display image corresponding to the nth frame of fusion image is different from the size of the shooting object in the second display image corresponding to the nth frame of fusion image, so that the image fusion module downsamples the first display image corresponding to the nth frame of fusion image to obtain a thumbnail image corresponding to the nth frame of fusion image, and the scale of the thumbnail image corresponding to the nth frame of fusion image is equal to the scale of the second display image corresponding to the nth frame of fusion image, which indicates that the size of the shooting object in the thumbnail image is the same as the size of the shooting object in the second display image. Finally, the image fusion module acquires parameters to be aligned and weight pairs corresponding to the N-th frame fusion image, and performs weighted fusion after center point alignment on the first display image corresponding to the N-th frame fusion image and the second display image corresponding to the N-th frame fusion image to obtain the N-th frame fusion image. It can be understood that when the alignment is performed based on the parameters to be aligned corresponding to the nth frame of fusion image, the center point of the second sending image corresponding to the nth frame of fusion image can be kept unchanged, and the center point of the thumbnail image corresponding to the nth frame of fusion image is reversely offset.

An embodiment of the application also provides an electronic device, which may include one or more processors and one or more memories. The one or more memories are coupled to the one or more processors and the one or more memories are configured to store computer program code that includes computer instructions that, when executed by the one or more processors, cause the electronic device to perform the functions or steps performed by the electronic device in the above-described method embodiments.

The embodiment of the application also provides an image processing device, which comprises a function/unit for executing the electronic equipment in the embodiment.

The present application also provides a chip system, as shown in fig. 8, the chip system 800 includes at least one processor 801 and at least one interface circuit 802. The processor 801 and the interface circuit 802 may be interconnected by wires. For example, interface circuit 802 may be used to receive signals from other devices (e.g., a memory of an electronic apparatus). For another example, interface circuit 802 may be used to send signals to other devices (e.g., processor 801). The interface circuit 802 may, for example, read instructions stored in a memory and send the instructions to the processor 801. The instructions, when executed by the processor 801, may cause the electronic device to perform the various steps of the embodiments described above. Of course, the system-on-chip may also include other discrete devices, which are not particularly limited in accordance with embodiments of the present application.

The present embodiment also provides a computer readable storage medium, where computer instructions are stored, which when executed on an electronic device, cause the electronic device to perform the functions or steps performed by the mobile phone in the above-described method embodiment.

The present embodiment also provides a computer program product which, when run on a computer, causes the computer to perform the functions or steps performed by the handset in the method embodiments described above.

In addition, the embodiment of the application also provides a device which can be a chip, a component or a module, and the device can comprise a processor and a memory which are connected, wherein the memory is used for storing computer execution instructions, and when the device runs, the processor can execute the computer execution instructions stored in the memory so that the chip can execute the functions or steps executed by the mobile phone in the embodiment of the method.

The electronic device, the communication system, the computer readable storage medium, the computer program product or the chip provided in this embodiment are used to execute the corresponding method provided above, so that the benefits achieved by the electronic device, the communication system, the computer readable storage medium, the computer program product or the chip can refer to the benefits in the corresponding method provided above, and are not repeated herein.

From the foregoing description of the embodiments, it will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of functional modules is illustrated, and in practical application, the above-described functional allocation may be implemented by different functional modules according to needs, i.e. the internal structure of the apparatus is divided into different functional modules to implement all or part of the functions described above.

In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another apparatus, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and the parts displayed as units may be one physical unit or a plurality of physical units, may be located in one place, or may be distributed in a plurality of different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated unit may be stored in a readable storage medium if implemented in the form of a software functional unit and sold or used as a stand-alone product. Based on such understanding, the technical solution of the embodiments of the present application may be essentially or a part contributing to the prior art or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, including several instructions for causing a device (may be a single-chip microcomputer, a chip or the like) or a processor (processor) to perform all or part of the steps of the methods of the embodiments of the present application. The storage medium includes a U disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, an optical disk, or other various media capable of storing program codes.

Finally, it should be noted that the above-mentioned embodiments are merely for illustrating the technical solution of the present application and not for limiting the same, and although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications and equivalents may be made to the technical solution of the present application without departing from the spirit and scope of the technical solution of the present application.

Claims

1. An image processing method, the method comprising:

In the working process of the first camera, receiving the operation of switching from the first camera to the second camera;

acquiring M frames of first images acquired in real time by the first camera and M frames of second images acquired in real time by the second camera after the operation is received, wherein M is a positive integer greater than 1, and N is a positive integer greater than or equal to 1 and less than or equal to M;

performing image fusion based on the M-frame first image and the M-frame second image to generate an M-frame fusion image, wherein the Nth frame fusion image is obtained by fusing the Nth frame first image and the Nth frame second image;

And displaying the M frames of fusion images based on the images acquired by the second camera.

2. The method of claim 1, wherein the generating the M-frame fused image based on the M-frame first image and the M-frame second image comprises:

obtaining a target fusion image based on a first display image corresponding to the target fusion image, a second display image corresponding to the target fusion image and a weight pair corresponding to the target fusion image;

The first sending and displaying image corresponding to the target fusion image is obtained after image processing is carried out on the target first image, the second sending and displaying image corresponding to the target fusion image is obtained after image processing is carried out on the target second image, the target fusion image is one frame of image in the M frames of fusion images, the target first image is one frame of image corresponding to the target fusion image in the M frames of first images, and the target second image is one frame of image corresponding to the target fusion image in the M frames of second images.

3. The method according to claim 2, wherein the obtaining the target fusion image based on the first display image corresponding to the target fusion image, the second display image corresponding to the target fusion image, and the weight pair corresponding to the target fusion image includes:

Acquiring a weight pair corresponding to the target fusion image under the condition that a first view angle corresponding to the first camera is larger than or equal to a second view angle corresponding to the second camera;

And carrying out weighted fusion on a first display image corresponding to the target fusion image and a second display image corresponding to the target fusion image based on the weight pair corresponding to the target fusion image to obtain the target fusion image, wherein the scale of the first display image is the same as that of the second display image, and the center point of the first display image is the same as that of the second display image.

4. A method according to claim 3, wherein the pair of weights corresponding to the target fusion image comprises a first weight corresponding to the target fusion image and a second weight corresponding to the target fusion image;

the weighting fusion is performed on the first display image corresponding to the target fusion image and the second display image corresponding to the target fusion image based on the weight pair corresponding to the target fusion image, so as to obtain the target fusion image, including:

weighting a first display image corresponding to the target fusion image based on a first weight corresponding to the target fusion image to obtain a first weighted image;

weighting a second display image corresponding to the target fusion image based on a second weight corresponding to the target fusion image to obtain a second weighted image;

And fusing the first weighted image and the second weighted image to obtain the target fused image.

5. The method according to claim 2, wherein the obtaining the target fusion image based on the first display image corresponding to the target fusion image, the second display image corresponding to the target fusion image, and the weight pair corresponding to the target fusion image includes:

Under the condition that a first view angle corresponding to the first camera is smaller than a second view angle corresponding to the second camera, downsampling a first display image corresponding to the target fusion image to obtain a thumbnail image corresponding to the target fusion image, wherein the scale of the thumbnail image is the same as that of a second display image corresponding to the target fusion image;

Acquiring parameters to be aligned corresponding to the target fusion image;

Based on the parameter to be aligned, aligning the thumbnail image to the center point of the second display image corresponding to the target fusion image to obtain an aligned image;

And carrying out weighted fusion on the aligned image and the second display image based on the weight pair corresponding to the target fusion image to obtain the target fusion image.

6. The method of claim 5, wherein the pair of weights for the target fusion image comprises a first weight for the target fusion image and a second weight for the target fusion image;

the weighting fusion is performed on the aligned image and the second display image based on the weight pair corresponding to the target fusion image to obtain the target fusion image, which comprises the following steps:

weighting the alignment image based on a first weight corresponding to the target fusion image to obtain a first weighted image;

7. The method according to claim 5 or 6, wherein the obtaining the parameters to be aligned corresponding to the target fusion image includes:

The method comprises the steps of obtaining a total alignment parameter and a completed alignment parameter corresponding to a target fusion image, wherein the total alignment parameter is used for indicating a center point difference value of a target first image and a target second image, the completed alignment parameter is used for indicating a parameter with a completed center point offset when the first display image and the second display image are obtained, and the value of the completed alignment parameter is smaller than that of the total alignment parameter;

And determining the parameters to be aligned corresponding to the target fusion image based on the difference value between the total alignment parameter and the completed alignment parameter.

8. The method of claim 4 or 6, wherein the first weight corresponding to the nth frame of fused image is greater than the first weight corresponding to the n+1th frame of fused image, and the second weight corresponding to the nth frame of fused image is less than the second weight corresponding to the n+1th frame of fused image, N being a positive integer greater than or equal to 1 and less than M.

9. An electronic device comprising one or more processors and one or more memories, wherein the one or more memories are coupled to the one or more processors, the one or more memories for storing computer program code comprising computer instructions that, when executed by the one or more processors, cause the electronic device to perform the method of any of claims 1-8.

10. A chip system for an electronic device, wherein the chip system comprises at least one processor and an interface for receiving computer instructions and transmitting the computer instructions to the at least one processor, and wherein the at least one processor executes the computer instructions to cause the electronic device to perform the method of any one of claims 1-8.

11. A computer readable storage medium, having stored therein computer instructions which, when run on an electronic device, cause the electronic device to perform the method of any of claims 1-8.