IES86537B2 - Improved smartphone imaging using an external peripheral - Google Patents

Abstract

An improved smartphone photographic experience is provided by using an external imaging peripheral. The peripheral is connected to the smartphone via a USB-3, or similar high-speed data bus and is configured to capture RAW image data of a higher quality than is possible with the smartphone's camera. The smartphone can be configured to disconnect its own camera and accept images from the peripheral over the high-speed data bus. A user interface displayed o the touch-screen of the smartphone enables the peripheral to be controlled, specifically the acquisition of a next image or activating a video recording. Additional control functions may include adjustment of the acquisition settings on the peripheral. Images from the peripheral are analyzed, processed and enhanced on the smartphone. Thus advanced image processing techniques available on the smartphone can be applied to images obtained from the peripheral. Images may be displayed, stored and transmitted over a network by the smartphone. <Figure 8>

Description

$86537 IMPROVED SMARTPHONE IMAGING USING AN EXTERNAL PERIPHERAL.

Field The invention is in the field of consumer imaging, more specifically an imaging add-on device for a smartphone.

Background Modern smartphone devices have settled on a compact and thin format. While this is convenient for the user, both in terms of carrying the device in a pocket and holding it in the hand while in use as a mobile phone it creates difficulties in resolving a high-quality optical image. Thus while the camera modules in today's smartphones have continued to improved in term of pixel resolutions, speed of image acquisition and a wide range of digital manipulation of the underlying image the optical quality is constrained by the physical limitations of the device form factor and the corresponding size of image sensor that can be accommodated within the device.

Some attempts have been made to improve the image quality by providing additional lens elements that clip onto the device over the existing camera lens to increase the zoom, enhance the field-of-view, or to provide improved macro capabilities. However these optical addon lenses are constrained to the design parameters of the original lens system and the size of the internal sensor element of the smartphone (Dainty 2012).

Some manufacturers have created independent camera modules that communicate with a smartphone via wireless communications such as Wifi (IEEE 802.11 ). These modules can be attached to the case of a smartphone or may be operated completely independently of the smartphone. They export a control user inferface (UI) for the camera and images are captured in the camera module and compressed to JPEG format (or MPEG format for video) prior to being transferred wirelessly to the controlling smartphone unit.

While these devices can obtain high-quality compressed images and video and may be controlled and operated from the smartphone, they do not support the acquisition of high quality RAW (Bayer) images. Furthermore they require a high-end full imaging system, including dedicated image signal processor (ISP) and main system CPU/GPU and JPEG (images) or MPEG (video) compression engine to be provided within the independent camera module.

With current state of prior art we see that improved imaging can be achieved on handheld imaging devices such as smartphones either by adding (i) an enhanced clip-on lens, or (ii) by connecting a dedicated and fully-independent camera module to the smartphone over a wireless (or wired) link. The first solution is limited by the original optical design and image sensor of the smartphone. The second approach overcomes these limitations but requires a full imaging pipeline and compression engine in order to process and compress images that are suitable for transfer over a wireless link.

Thus there is a need for an add-on peripheral that improves the 10 optical acquisition, and can accommodate a larger sensor size (APC or full-frame) but that can also take advantage of the inbuilt image processing capabilities and system-level 'apps' of the smartphone.

Description of the Drawings Figure 1 illustrates the RAW camera peripheral Module with the main 15 sub-components required for various embodiments.

Figure 2 illustrates the various functions of an example Image Signal Processor pipeline showing partitioning of functionality to allow more energy efficient operation (from US 2013/0004071). The present invention employs similar partitioning to allow separation of the RAW (Bayer) processing from higher level image processing functions.

Figure 3(a): Shows a MIPI interface as implemented inside a smartphone. MIPI provides an interface between camera-sensor and ISP; Figure 3(b): Shows the same smartphone connected to the RAW camera module via a USB-3 interface; the smartphone has access via USB-to2 5 MIPI to RAW module SIP and sensor subsystems as if they were incorporated into the smartphone.

Figure 4(a) & 4(b): Camera module with USB connector and full-size lens module.

Figure 5: Smartphone configuration to interface with camera module.

Figure 6: Exemplary smartphone interface used to control camera settings (ISO, EV number, aperture and 'equivalent' exposure time) available in smartphone camera app.

Figure 7(a) & (b): RAW module with interchangeable hand-grip; and configured to receive smartphone in a landscape orientation.

Figure 8: Exemplary Smartphone User Interface for RAW Camera Peripheral.

Brief Description of the Invention For a better understanding it is useful to consult US 2013/0004071 Image signal processor architecture optimized for low-power, processing flexibility, and user experience to Chang et al. This describes an image signal processor architecture that may be optimized for low-power consumption, processing flexibility, and/or user experience. In one embodiment, an image signal processor may be partitioned into a plurality of partitions, each partition being capable of entering a lower power consumption state. Techniques for the partitioning of an ISP as described in this patent application may be used advantageously in the present invention.

An example ISP is shown in figure 2, taken from this patent application. It can be seen that the ISP as shown in figure 1 is significantly more complicated. In a practical embodiment the first partition of the ISP shown in figure 2, that is 202, would be equivalent to the Bayer ISP of figure 1. The other functional elements of the ISP, that is 204 and 206, 208 would be implemented on the host smartphone.

Figure 3 shows the MIPI interface between a sensor and ISP, taken from www.mipi.org and based on CSI-2 variant. The latest practical embodiments are known as M-Phy™ (physical layer) and UniPro™ (protocol stack). US-2013/0191568 Operating m-phy based communications over universal serial bus (usb) interface, and related cables, connectors, systems and methods to Hershko et al. describes how M-Phy (MIPI) interfaces can be controlled and data transfers achieved via a USB-3 interface.

Thus a physical example of the present invention is presented in figure 4 which illustrates the camera module with external USB interface. A smartphone with external USB connector can be attached on top of said module and the module is then operated and controlled via the USB interface. The internal details of the camera module are illustrated in figure 1 and the internal architecture of an exemplary smartphone configured to connect to the device is shown in figure 5.

Various detail of this invention and alternative embodiments will be documented in the following sections.

Detailed Description of the Invention In a preferred embodiment the camera module is configured with a full-size DSLR or APS-C image sensor and incorporates a lens mount compatible with a DSLR manufacturer. This allows the module to use any lens design from that particular camera maker. In addition the module will provide various electrical and mechanical interfaces as required to operate the lens type(s) of that manufacturer. For example, US 8452170 Digital single lens reflex camera to Hwang et al describes details of one such exemplary interface. This document, US 8452170, is incorporated by reference.

The camera module will also include exposure, auto-focus, shutter (capture) and zoom control subsystems compatible with the lens types of the relevant manufacturer.

The RAW Camera Module io Physical Embodiments The main embodiment is shown in figure 4(a) with a rear-view in figure 4(b). An alternative embodiment is shown in figure 7.

The RAW camera module comprises a main body [401] which is of greater depth than the smartphone and is provided with clamps, bindings or other fixing mechanisms (not shown) to facilitate connecting the smartphone to the module in a robust manner. The smartphone will be connected electrically to the module via a USB-3 or similar highspeed data port [405] and in the embodiment shown in figure 4(a) will be mounted in a portrait orientation.

In certain embodiments the RAW module also features a lens mount [407] that facilitates interchangeable lenses [409]. Electrical connectors are also provided in the lens mount to enable on-lens switches and controls to provide settings information to the lens control subsystem [109] of the RAW module (shown in figure 1).

In certain embodiments the RAW module includes a hand grip (figure 7(a)) that improves handing of the device. This handgrip may be detachable so that it can be switched from left-hand to right-hand side and thus facilitate use of the module by left-handed photographers. In other embodiments the module may be configured to allow the smartphone device to be connected in a landscape orientation allowing a widescreen user-interface to be provided.

Internal Organization of the Module This is shown in Figure 1. A lens [110] is typically provided as a 35 separate component that can be removed and replaced with another lens with different properties. As with digital single lens reflex (DSLR) cameras this allows a variety of wide-angle, macro and telephoto lenses to be used to acquire images in different photographic situations. As such lenses are relatively large it does not make sense to adapt a smartphone to accommodate removable lens assemblies, but the RAW module enables the use of such lens assemblies when the smartphone is connected.

In addition to the lens assembly the RAW module also incorporates a 5 lens control [109] (and optionally a lens driver/motor, not shown) and focus and exposure control subsystems [114], The purpose of the lens control is to drive the movable lens elements as required by the focus subsystem.

In some embodiments a phase focus subsystem may be used and this may include an additional sensing element. Alternatively a simple contrast based focusing system may be employed and the focus subsystem will use either RAW data direct from the image sensor, or may use the ISP to pre-process certain RAW image data. In any case the contrast based focus subsystem uses selected portions of the full image scene and measures the contrast levels within these regions to determine the effects of changing the lens position. A range of other focus sensing approaches are described in the literature and may be adopted here.

The lens assembly may also include an aperture ring that allows changing the size of aperture that admits light into the camera and onto the image sensor [112], It is also possible to adjust the time interval during which image data is accumulated by the pixels of the imaging sensor. Typically this will be a CMOS sensor and controlling the combination of lens aperture and the accumulation interval for image data by pixels of the sensor is equivalent to exposure control. As both focus and exposure control subsystems interact with the lens assembly, the imaging sensor and potentially the Bayer-ISP these are shown as a single system block [114] in Figure 1.

The image sensor [112] is typically a CMOS image sensor. This comprises a pixel array ranging in size from 5 Megapixel with the largest in use in consumer products being of the order of 40 Megapixels. Also the size of this sensor is important as higher image quality is obtained by using a larger image sensor. In smartphones a relative small sensor is typically used — this can be as small as 2 25sq mm (25mm ) up to just over 100sq mm (100mm );large sensor sizes that are more suitable for the RAW peripheral include 4/3 sensors (225mm2), APS-C (320-370 mm2) or APS-H (550 mm2).

In addition to a larger sensor the RAW Module also includes a specialized image signal processor (ISP) unit [105]. This specialized module is optimized to accelerate processing of the RAW image data and in a typical embodiment will provide the various Bayer processing functions described by [202] of Figure 2. In some embodiments this processed Bayer data is transmitted directly to the smartphone but in alternative embodiments it may be further processed to YUV, YCC or RGB color spaces. The provided color space and the pre-processing provided will depend on an initialization process when the smartphone is first connected to the module. In some embodiments it may be possible to change this configuration from the connected smartphone, or by means of a mode switch on the module, if provided.

The transmitting of an image frame to the connected smartphone is managed by a MIPI to USB-3 interface subsystem [116].

The Image Processing Pipeline This typically refers to a range of image processing that is applied to a RAW (Bayer pattern) image that is obtained from an image sensor. The Bayer pattern is well known to those skilled in the art and employs additional sensing elements that are responsive to green wavelengths.

The origins of this nomenclature stretch back to Bryce Bayer's patent (U.S. Patent No. 3,971,065) in 1976 where the inventor refers to the green photosensors as luminance-sensitive elements and the red and blue ones as chrominance-sensitive elements. He used twice as many green elements as red or blue to mimic the physiology of the human eye. The luminance perception of the human retina uses M and L cone cells combined, during daylight vision, which are most sensitive to green light. These elements are referred to as sensor elements or pixel sensors, or simply pixels; sample values sensed by them, after interpolation, become image pixels.

The raw output of Bayer-filter cameras is referred to as a Bayer pattern image. Since each pixel is filtered to record only one of three colors, the data from each pixel cannot fully specify each of the red, green, and blue values on its own. To obtain a full-color image, various demosaicing algorithms can be used to interpolate a set of complete red, green, and blue values for each pixel. These algorithms make use of the surrounding pixels of the corresponding colors to estimate the values for a particular pixel.

Note that different camera manufacturers have arranged the storage of Bayer data in different, proprietary, file formats. Also, some cameras use variations on the original Bayer color filter array with different sensor pixel distributions, or use additional sensor pixels of enhanced sensitivity. The generic term of RAW image data is widely used in the digital imaging industry to refer to the unprocessed sensor data. Throughout this document when reference is made to Bayer data or RAW image data the intended scope is any data captured at the full data resolution {typically 12- or 14-bit) as read out from each of the camera's image sensor pixels and not the narrower scope as defined in U.S. Patent No. 3,971,065.

Now, returning to our description of the image-processing pipeline, after the demosaicing process a conventional RGB image is provided but this still requires additional processing. More specifically the adjustment of the image to provide gamma correction, white balance and color tone balance. Each of these image adjustments are scene dependent and determining and applying each requires some computation based on statistical analysis of the acquired image.

The image is then typically converted into YCC or YUV color space as this facilitates further manipulation of image luminance (Luma) and color (Chroma) and eventually image compression to JPEG format. An exemplary image processing pipeline is illustrated in figure 2. In a modern digital imaging device the bulk of the processing pipeline [106] is implemented in a dedicated processing unit known as an image signal processor (ISP). The use of a dedicated processing unit allow for the use of dedicated hardware functions that would normally be provided by software. These dedicated hardware functions can process image data at significantly higher rates than a pure software implementation.

In Figure 2 the ISP pipeline comprises several distinct elements. First the sensor data [104] is input to a first Bayer data processing unit [202] that performs a number of exemplary functions including compensating for defect pixels on the sensor, correcting fixed pattern noise, adjusting for lens shading (luminance decreases towards outside of lens), calculating image statistics (e.g. histograms) and Bayer scaling. This modified Bayer data is next passed to a demosaicing and conversion block [204] that generates first a RGB image, then after adjustment of the RGB it further converts the image to YCC or YUV format. A third block [206] performs further adjustments on the YUV source data and a fourth block is often present and allows image re-scaling [208] to provide, for example, a preview stream to display on the screen of the imaging device.

In state-of-art imaging devices all four main processing blocks, [202], [204], [206] and [208] are provided in a single system-on-chip or application specific processor known generically as an image signal processor (ISP).

In a state-of-art imaging device it is likely that further processing (e.g. high dynamic range (HDR), specialized filters, face and eye tracking, smile detection or red-eye filter) will be performed on the main CPU/GPU of the device. As these algorithms are frequently tuned and adjusted they are more suited to a software implementation rather than providing dedicated hardware.

This leads to the underlying inventive concept which is to split the 10 Bayer and lower-level image processing into a separate peripheral and by bypassing the internal sensor and ISP of the smartphone and providing RAW image data (or RGB/YUV/YCC in some embodiments) directly to the smartphone from an external peripheral optimized to provide high-quality optical and sensor subsystems similar to those found in DSLR or high-end mirrorless cameras.

Internal Organization of the Smartphone This is shown in figure 5. The smartphone contains a camera module [503] that is typically designed independently of the main phone. It will typically be interfaced with the main device using MIPI (M-Phy physical interface) and in most modules only the lower level Bayer processing corresponding to [202] in Figure 2 is provided within the module.

Other processing corresponding to [204], [206] and [208] is typically performed in a dedicated ISP component [516] and a pre-processed RGB or YCC/YUV image is passed on to the main CPU/GPU [509] for more complex analysis and post-processing. Where such processing is performed it is typically required to store the image, or portions thereof, in a temporary image store [511] and image portions may be moved back and forth via internal data bus [518] between the main CPU/GPU [509] and this image store. Once processing of the YCC/YUV/RGB image is finalized it is sent to the compression module [505] and permanent storage [507].

Where additional analysis and processing is not required the image may simply be sent via internal data bus [514] for compression [505] and permanent storage [507].

Returning to the smartphone camera module [503] and ISP [516] we note that the standard for external data connections on mobile devices is typically the USB-3 standard. Many smartphone devices incorporate a USB-3 module and can be interfaced with external USB-3 peripherals either directly, or using a specialized adapter.

Now US-2013/0191568 Operating m-phy based communications over universal serial bus (usb) interface, and related cables, connectors, systems and methods to Hershko et al. describes how M-Phy (MIPI) interfaces can be controlled and data transfers achieved via a USB-3 interface.

Thus it will be understood that the ISP [516] in figure 5 which normally accesses the camera module [503] using MIPI interface to obtain digital images could be modified to issue a similar set of MIPI commands to a remote device over the USB-3 bus [501] and to accept digital image data from a remote source connected to the same USB-3 bus. A detailed description of such technique is given in US2013/0191568, hereby incorporated by reference.

Thus in a preferred embodiment when the RAW camera peripheral is connected to the smartphone the ISP is notified and can be configured to access the RAW peripheral rather than the internal smartphone camera module. This provides high quality RAW images to the smartphone ISP and after further processing to the main CPU/GPU and compression module for eventual storage [507].

This is further shown in figures 3(a) and 3(b).

Figure 3(a) shows the connection between the smartphone ISP [318] and the Smartphone camera module [503]. The CCI master [311] establishes communication with the CCI Slave [303] and when the appropriate control sequence is transmitted the main MIPI interface between CSI Transmitter [305] and CSI receiver [314] can be activated. This enables data to be transmitted at high speed via the parallel data buses DATA-1+/- to DATA-N+/-. These utilize communications to allow faster data transfer possible using serial bus communications. The received data is transferred to the ISP central core [307] where it is stored temporarily while being processed (if required) by the ISP core. The data is then transmitted to the main application processor of the smartphone via the main smartphone system bus [316]. The USB interface [501] is unused in this configuration. differential line than is normally Figure 3(b) shows how the same smartphone is configured to use the external RAW Camera Module. In this configuration the internal MIPI bus of the smartphone is mostly disabled (although buffer registers may remain active as will be explained shortly) and the communications functionality of the CSI Receiver [314] and CCI Master [311] are inactive. Communication is now effected via the USB-3 module [501] of the smartphone which is preferable integrated with the smartphone ISP [318]. The ISP core [307] can transmit control sequences to the MIPI-to-USB module [116] of the RAW Camera Module [323] where it is converted into MIPI signals sent over the internal MIPI bus of the RAW Camera Module. This, in turn, can activate any of the MIPI connected modules, the RAW sensor [112] which is behind the RAW ISP [105]. The RAW ISP [105] can be configured to be transparent to provide access to the sensor. Alternatively it can be programmed to perform a range of Bayer Data Processing [202] as shown in figure 2. The RAW Image Store, where provided, may also be accessed and can buffer one or more image frames depending on the size of image frame and the memory provided in the RAW camera module. These stored image frames can be used to assist image pre-processing and may also be used for HDR type functions. After control settings are programmed on the RAW Camera Module then image frame acquisition is initiated with an 'acquire next image frame' instruction to the module [323]. Data from the image sensor (or ISP if it is actively processing data) is clocked over the parallel differential data channels of the MIPI interface into the MIPI-to-USB module where they are interleaved into a single packetized USB data stream. This is received by the USB interface of the Smartphone ISP [501] and typically the data is deinterleaved into the corresponding buffer registers (not shown) of the onboard MIPI bus as if it were received from the original onboard MIPI. The ISP Central Core [307 ] may now commence to process the received USB data as if it had originated on the original Smartphone device [329].

We next explain how manual controls on the RAW peripheral can be used to set and configure the acquisition settings in a number of embodiments.

Manual Controls In certain embodiments manual controls may be added to the body of the module to enable manual adjustment of these subsystems, however they may also be operated from the smartphone UI as will be explained shortly. Where manual controls are provided an additional microcontroller (not shown) may be added to determine and interpret the state of various manual settings and may determine other settings based on look-up tables (LUTs) or calculated from pre-programmed formulae. Where provided this microcontroller will also store settings, typically on a dedicated non-volatile storage and communicate these settings and the configured 'state' of the module to the module-ISP.

In turn the module ISP will make such information available to the connected smartphone via the MIPI-to-USB communications link. Thus where controls for manual settings are provided these will update the state of the module as communicated to the smartphone, which will, in turn, adjust its UI to reflect the changed state. In some embodiments a low-cost, low power LCD display may be provided and managed by the microcontroller. This enables the user to set and check acquisition parameters directly on the module without a need to consult the smartphone display.

In some embodiments a selection switch is provided that can disable the manual controls on the body of the camera module and allow it to be controlled directly from the smartphone UI and touchscreen. In alternative embodiments the decision to enable or disable manual controls may be determined by the model of smartphone that is connected to the peripheral.

The module may also provide an external video-mode, or video-capture switch.

A mode selection switch similar to that provided on modern cameras that allows the module to be switched into fully-automatic, semiautomatic {aperture or shutter priority or exposure EV number), and fully manual modes may also be provided.

In alternative embodiments the camera module may not provide manual controls, but can be entirely controlled from the touchscreen interface of the smartphone.

In some embodiments the peripheral may include un-numbered adjustment dials for shutter time, aperture, ISO settings or equivalent. These dials allow the values for each setting to be adjusted up(+) or down{-) and the result will be stored in the smartphone app. Thus a photographer may control settings as with a normal camera, using conventional dials even though the setting values are stored in the corresponding smartphone app.

Hardware Processing In a preferred embodiment the camera module contains a basic image signal processor that operates only on RAW Bayer images. Some dedicated hardware enhancements may be provided to support high quality image acquisition. Typically these will determine one or more characteristics of an image frame and apply the results to modify the acquisition of a following frame.

More specifically focus sharpness may be calculated and provided to an auto-focus subsystem; averaged pixel luminance across blocks or regions of the image may be determined and used as input to an auto40 exposure sub-system; frame to frame motion can be estimated using integral projection techniques (summing rows S columns of each image frame) and a combination of these techniques applied to sub-regions of the image frame can be used to determine frame-to-frame motions, both translational and rotational. When working on raw images the green channel is frequently used as equivalent to the luminance channel.

In certain embodiments more advanced hardware techniques may be employed. For example a basic face detector can be implemented using a small number of the most inclusive haar classifiers (Cho et al. 2009; Tresadern et al. 2011; Yang et al. 2010) and can provide information on regions of the image; by further separating such classifiers into symmetric and non-symmetric we can also find partial faces (Corcoran et al. 2013). However the hardware engines being incorporated in smartphones should be employed where practical, thus most of the hardware incorporated into the camera module is targeted to improve raw image acquisition. Refinement and enhancement of the acquired image will mainly be achieved by the smartphone ISP.

Camera Module Operating Modes The camera module ISP communicates with the image sensor over a MIPI interface to acquire a full image frame. This may be initiated by the smartphone, but may also be initiated by a manual shutter/capture button where provided. When a full capture has not been initiated the camera module defaults to a preview-stream mode, unless it has entered a sleep state.

Preview Stream Mode In the preview stream mode the module-ISP continually acquires a lower resolution image frame — typically matched to the smartphone UI — and transmits this to the smartphone over the USB-3 interface. If auto-focus is active the focus subsystem adjusts the focus position from frame to frame to achieve the sharpest focus setting. If manual focus is active then the focus is only changed in response to adjustments of the attached lens. The preview stream from the camera module is further processed by the smartphone-ISP to provide an RGB/YCC image stream for display on the smartphone touchscreen. Other 'controls' may be provided on the touchscreen, allowing the user to manually adjust acquisition parameters and camera module settings.

The preview stream may be buffered on a temporary basis in the raw image buffer on the camera module. Typically this will be only for a limited number of image frames and after this fixed number of frames the buffer will be overwritten. However this allows a history of recent image frames to be retained and possibly used to enhance the current (latest) image frame without placing a memory load on the smartphone. Select portions of buffered image frames may be uploaded as required by image processing algorithms on the smartphone.

Metadata from the acquisition subsystems can be saved with the 5 corresponding image frames in the temporary image store. This includes focus, exposure and zoom data as well as time & date of acquisition.

In some embodiments it may include metadata associated with hardware enhancements provided on the camera module or built into the ISP.

For example, if a hardware face detector was provided this metadata might indicate if potential face regions were detected in an image and, if so, how many potential face regions were found (overlapping detections may be counted as just a single face) and the XY locations of potential face regions.

Other useful hardware functions might include flash-eye detection, calculation of an integral image, integral projection vectors, various image histograms, foreground/background and sharpness/contrast maps.

In some embodiments additional hardware subsystems such as GPS may be included and thus location data could also be provided.

Initialization The RAW camera module includes an independent power subsystem, and battery. It may operate in different embodiments as a USB master, or USB slave but in either case the device is self-powered. In certain embodiments it may include a substantial battery pack, serving as a reserve power source for a smartphone.

When the peripheral is first attached to the smartphone the two devices will exchange configuration information.

In some preferred embodiments the peripheral will be controlled from a dedicated camera app on the smartphone and this app will give access to low-level system features of both the smartphone, including direct access to the internal camera module interfaces of the smartphone and the peripheral via the USB to MIPI interface module.

In alternative embodiments control may be from external camera controls - similar to the control knobs provided on a typical DSLR camera — and the peripheral will communicate these settings to the attached smartphone (which may optionally display them). It is possible for the peripheral to provide functionality to operate in both modes, but not at the same time — thus some configuration of the peripheral, either by an external mode switch, or a non-volatile internal state may be needed.

After the two devices have completed this configuration phase the peripheral will typically be configured in its normal operational mode of image acquisition. The smartphone will also enter a custom configuration where the low-level image acquisition functions of its camera subsystem, specifically the optical and sensor components, are disengaged at the smartphone ISP interface, and the corresponding MIPI instructions are diverted instead to the USB-3 interface, The smartphone will typically control normal operational modes of the RAW peripheral through a dedicated camera 'app' or application, although these could also be exposed as an API at the OS level. In the latter case 3rd party 'apps' could access the API and thus directly control the RAW peripheral.

Image Acquisition Mode In this mode the various image acquisition settings are programmed on the peripheral as with any conventional digital camera. The peripheral will provide a fully automatic mode of image capture, one or more semi-automatic modes and a full manual mode. Optionally the peripheral may be programmed to provide more scene-specific 'smart' modes, but as these will typically require image post-processing and are controlled from the smartphone they are not considered here. (i) Full Auto Mode In the automatic mode the peripheral is configured to implement selfcontained auto-focus and exposure of the imaged scene. In this mode the peripheral control sub-systems and ISP continually analyze the acquired preview scene and choose the optimum shutter speed, aperture, ISO and flash settings for image acquisition.

Once set into this mode the peripheral activates and acquires images constantly and adjusts lens, exposure timing and ISP processing to optimize the acquired scene. Typically this may require some buffering of image frames to perform frame-to-frame calculations and thus the RAW image store is accessed by the ISP as required. In some embodiments the acquired image stream will be reduced from full resolution enabling greater control over the frame-to-frame processing and allowing multiple image frames to be buffered.

As the user requires to view the imaged scene in order to compose an image the acquired and partly processed (at Bayer level) preview image stream is transmitted over the USB-3 interface to the attached smartphone for display. Additional processing on the smartphone is limited to conversion from Bayer to YCC or similar format suitable for real-time display on the smartphone screen. Typically the user does not have direct control over focus, exposure, white balance, or any other camera parameters, but it may be possible in some embodiments to adjust parameters such as ISO to control the sensitivity of the 'auto' algorithms.

When the user decides to acquire an image this may be actuated on the smartphone, or by pressing a capture button on the peripheral. In the first case a command is sent via the USB to MIPI interface module to instruct the ISP to acquire a full resolution RAW image frame and transmit this to the smartphone via a MIPI to USB transfer. Any acquisition metadata may also be transferred together with status information immediately preceding or following the image transfer.

In some embodiments where a HDR mode is available a first frame will be acquired and transferred, camera settings may be changed and a second image frame is acquired and also transferred together with any relevant image frame metadata. If some Bayer level processing is employed as part of the process to create a combined HDR image then the first frame may be temporarily stored in the RAW image store on the peripheral. Image frame metadata from one or both image frames may optionally be transmitted. In certain embodiments more than two image frames may be acquired to create a HDR image. Typically each frame acquisition will be with different acquisition parameters, although this is not a requirement and in certain embodiments more than one frame with the same camera settings may be acquired as references images.

After transmitting to the smartphone the RAW image frames are further processed by the ISP of the smartphone camera subsystem. This process is essentially the same as if the image had been acquired by the smartphones camera and sensor, but the smartphone now has access to improved RAW images with better optical quality due to the use of a DSLR lens unit and a larger full-frame or APS-C image sensor. (ii) Semi-Automatic Modes These provides a mode found on DSLR cameras where the user may select a particular acquisition parameter, e.g. exposure time, aperture, etc. The most commonly found modes are: AV (Aperture-Priority), TV or S (Shutter-Priority) and P (Programmed Auto).

Aperture-Priority allows the photographer to set the aperture value and the peripheral automatically sets the correct shutter speed; TV lets the photographer choose the shutter speed first (for example when shooting sports) and the camera automatically sets the correct aperture. P-Program mode is similar to Auto mode - the shutter and the peripheral determines aperture settings, but the photographer can adjust the shooting and image-recording functions.

Alternatively, some of these modes are sometimes presented as being suitable for specific scene contexts or shooting conditions. As examples: Portrait Mode (P): In this mode the peripheral subsystems are configured to assume a subject in the foreground of the frame and choose a shallow depth of field in order to keep the human subject in focus but the background blurred. In low-light situations a flash may be activated.

If the peripheral reads the scene as dark, it may be able to add fill-in flash, increasing the luminance of certain regions of the image. This must be implemented at Bayer level and may not be feasible. In particular, it is best applied selectively to facial regions and optionally to strongly shadowed foreground regions. However with implementation of a basic face-detection or foreground mapping as part of the ISP hardware this can be more effectively implemented.

Landscape mode: Typically uses a small aperture (high f/number) to create a well-focused image from the foreground into the distance. Landscape mode tends to suit a wide lens, and again works well if the scene is well lit. Flash is normally disabled.

Sports Mode: Because sports are fast paced activities, sports mode uses high shutter speed of at least 1/500 — 1/1000 of a second. With a high shutter speed to freeze movement, it means that flash is normally disabled and a brightly lit scene is required. Sports mode can work well alongside continuous shooting mode, where images are taken consecutively — the result is, for example, a number of shots capturing action in mid air.

Might Portrait: In the night portrait mode, the peripheral control systems and ISP try to balance the darkness of the background with the need to light the subject in the foreground. The aperture will have to be wide to allow sufficient light to capture the background and keep the subject in focus, but at the same time flash is required to illuminate the subject and avoid blur. Sometimes the night portrait mode will double flash, creating an unusual double exposure look.

These 'semi-automatic' modes are distinguished from 'smart-scene' modes where additional, post-acquisition, image processing is required. Often this employs multiple image frames. Some modern smartphones and cameras have a very long list of such 'smart' modes.

In a sense, two-image HDR is an example of such a 'smart-scene' but because it has become so prevalent in modern smartphones it makes sense to modify the imaging peripheral and Bayer ISP to accommodate this particular form of image acquisition. (iii) Manual mode In full manual mode the RAW peripheral does not make any attempts to adjust settings. Instead these are set manually — either by external switches and dials on the peripheral itself, or from the smartphone UI. An example of a manual UI for a smartphone is shown in figure 6. This enables the user to directly set the ISO, white balance, exposure time and aperture (f-stop) of the camera. Manual adjustment of focus, noise suppression and optical stabilization may also be provided where such subsystems are incorporated into the RAW peripheral.

In response to such settings the peripheral will acquire and process RAW images and transmit these to the smartphone, thus the user can see the effects of the various settings from the preview stream transmitted from peripheral to smartphone. The exact settings may vary for different embodiments, but will mirror those available on today's DSLR cameras. (iv) 'Smart' Scene Modes (with post-processing) These will typically require multi-frame image post-processing and are thus controlled from the smartphone. In such modes the smartphones would simply manipulate the peripheral to capture multiple sequential image frames, possibly with different acquisition setting as described below.

Some examples include: Best Face Mode: This captures multiple image frames of a group of people and allows the user to select the best face from each; typically it captures at least 5 consecutive image frames.

Beauty Modes: This modifies a portrait image and is mainly based on image post-processing, although some features may require two or more consecutive image frames to be acquired with different camera settings on the peripheral.

Bokeh and Smart-Portrait Modes: Bokeh mode blurs the background in a controlled manner and normally it is sufficient to acquire two consecutive image frames with different focus settings. Some 'smart40 portrait' modes can provide Bokeh-like blur without a need for the computationally complex blurring required to simulate lens blur — these typically require 3 consecutive image frames to be acquired. Night Portrait or Low-Light Modes: Basic night portrait mode uses controlled exposure and camera flash to capture a single image, but it can be improved using separate image frames with flash and noflash settings and combining the two.

A range of other flash/no-flash techniques are described in, for example, Digital Photography with Flash and No-Flash Image Pairs by Petschnigg et al from Microsoft Research and available at: http://research.microsoft.com/en-us/um/redmond/projects/flashnoflash/ Another low-light technique involves capturing a blurred image with full exposure and an unblurred image with short-exposure time. The two images can be combined to generate a sharp low-light image.

Focus Stacking or Refocus: Another technique is to acquire multiple images with different focus setting and combine the 'in-focus' parts of each image. In other examples these may be stored in a container file providing a means to re-focus an image.

HDR De-Ghosting: Motion that occurs between the two HDR image frames leads to a ghostly effect where a face, or body has moved between the two image frames. It is possible to remove such 'ghosts' by careful and advanced post-processing but this will be computationally expensive and would not be implemented on the RAW peripheral described here.

In such 'smart' modes the main computation and post-processing will be implemented on the smartphone, which will typically incorporate a multi-core CPU and GPU units to facilitate such advanced image processing algorithms. The peripheral does not need to provide any of the processing for such mode, only to provide the raw images with appropriate settings as requested by the smartphone.

Video Acquisition Mode Video mode may be selected via the smartphone, or from a switch on the RAW peripheral. Once this mode is activated the peripheral switches to continuous acquisition and transfer of image frames to the smartphone. The Bayer ISP is set into a fixed configuration (e.g. ISO, noise suppression and Bayer processing modes) although it may still be adjusted by the focus and exposure subsystems. And the ISP can continue to provide basic image frame analysis (histogram & focus feedback) to these subsystems as in the auto-camera mode.

In a preferred embodiment full resolution image frames are transmitted to the smartphone over USB-3 at 30-60 frames per second.

Post-processing and compression to an MPEG stream is delegated to the smartphone which uses the same compression engine as it would for native images. If the video resolution supported on the smartphone is less than that provided by the peripheral then 'full resolution' from the peripheral may be re-sized to the smartphone resolution before transfer over USB-3.

It is envisaged that at least 1080p HD video would be provided, but 4K video is more likely to become the standard for smartphones as high-speed LTE networks are deployed. In any case the video capabilities of the peripheral and smartphone would be part of the initialization process described previously.

Smartphone Workflows The peripheral is primarily designed to substitute for the inbuilt camera of the smartphone at the RAW level. Thus it is optimized for the acquisition and Bayer-level processing of image sensor data and to provide DSLR equivalent focus, exposure and image frame calibration capabilities for DSLR or APS-C sized image sensors.

Image transfers are optimized to minimize transfer lag, and image data can be transferred directly to the host smartphone with only a small lag as the image bitstream passes through the Bayer ISP on the peripheral.

These raw images are processed on the smartphone using the phone ISP and treating the Bayer images as if they had originated from the smartphone camera subsystem. Thus all of the advanced smart-scene processing can be available on the smartphone, but using higher quality RAW images due to the DSLR quality optics and sensor. Accordingly, in figure 8 we show an alternative user interface corresponding to the vertical alignment of the smartphone. A 'liveview' preview of the imaged scene is provided on the smartphone screen. This is streamed directly from the RAW module and may be optionally subsampled on the RAW module to reduce resource requirements on the smartphone device.

Smartphone User Interface As mentioned above, most of the higher level imaging capabilities of the smartphone can still be used, taking advantage of higher image quality obtained from the RAW module. However, in order to correct set the acquisition parameters for said module when operating in semi-automatic or full manual modes it is necessary to provide the user with simplified access to these settings.

In this regard figure 6 provides an example of a typical user interface in use by hybrid camera devices such as the Samsung Galaxy camera or the Nokia 818 or 1020 camera-phones.

Note the mode selection switches to the right that provide a standard range of modes used in DSLR or mirrorless cameras. These are summarized by table 1 below: Mode Shutter Speed Aperture P (programmed auto) Selected by camera Selected by camera S (shutter-priority auto) Selected by photographer Selected by camera A (aperture-priority auto) Selected by camera Selected by photographer M (manual) Selected by photographer Selected by photographer Table 1: Acquisition inodes for a typical DSLR camera.

It can be seen that these are quite complex and would be found confusing by a novice user. The different modes are activated by selecting the appropriate switch, for example the 'S' mode or shutter-priority can be actuated by switch [609]. The current selected mode is highlighted in the user interface, in the example of figure 6 it is the 'P' mode, or programmed auto switch [611].

Depending on the selected mode and according to table 1 above the user will have access to some of the camera acquisition setting dials, namely ISO [603], EV or exposure number [605], f-stop or aperture size [607] and shutter speed [614]. The settings that are autoselected by the camera can not be changed and are typically greyedout or there user interface elements are shown as inactive. For an inexperienced user this can be quite confusing. A further problem is that the user does not have direct access to the preview of the scene when setting acquisition parameters from a user interface as shown in figure 6 .

It will be seen from figure 8 that all of the main camera acquisition parameters are provided [814] as rotary-dial switched as commonly found on smartphone user interfaces (e.g. to set alarm times). The user may conveniently flick through settings and observe the effects on the 'live-view' image stream. In addition different modes may be selected as shown to the right with S-mode [809], P-mode [802], Amode [807] and full manual [811] available. A dedicated video mode is also provided [804], Typically, in video mode there may be less control over image acquisition, although this will depend on the underlying capabilities and computational resources available both within the RAW module and the connected smartphone.

In addition a range of programmable 'f-buttons' [816] may be provided. These can provide access to a range of advanced filters available on the smartphone. Examples include image analysis functions such as red-eye filter, face tracking, beauty filters, smile or blink detection, and so on. These may be programmed by the user, or some may be provided with a dedicated function, e.g. flash on/off or redeye on/off. In some embodiments these switched could cycle through multiple modes of a filter, or select how to combine two or more filters that are commonly used together.

Those of skill in the art would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithms described in connection with the embodiments disclosed herein may be implemented as electronic hardware, instructions stored in memory or in another computer-readable medium and executed by a processor or other processing device, or combinations of both. The arbiters, master devices, and slave devices described herein may be employed in any circuit, hardware component, integrated circuit (IC), or IC chip, as examples. Memory disclosed herein may be any type and size of memory and may be configured to store any type of information desired.

To clearly illustrate this interchangeability, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. How such functionality is implemented depends upon the particular application, design choices, and/or design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a processor, a DSP, an Application Specific Integrated Circuit (ASIC), an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

The embodiments disclosed herein may be embodied in hardware and in instructions that are stored in hardware, and may reside, for example, in Random Access Memory (RAM), flash memory, Read Only Memory (ROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of computer readable medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a remote station. In the alternative, the processor and the storage medium may reside as discrete components in a remote station, base station, or server.

It is also noted that the operational steps described in any of the exemplary embodiments herein are described to provide examples and discussion. The operations described may be performed in numerous different sequences other than the illustrated sequences. Furthermore, operations described in a single operational step may actually be performed in a number of different steps. Additionally, one or more operational steps discussed in the exemplary embodiments may be combined. It is to be understood that the operational steps illustrated in the flow chart diagrams may be subject to numerous different modifications as will be readily apparent to one of skill in the art. Those of skill in the art would also understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves magnetic fields or particles, optical fields or particles, or any combination thereof.

The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

References Cho, J. et al·., 2009. Fpga-based face detection system using Haar classifiers. In Proceeding of the ACM/SIGDA international symposium on Field programmable gate arrays - FPGA '09. New York, New York, USA: ACM Press, p. 103. Available at: http:I/dl.acm.org/citation,cfm?id=1508128.1508144 [Accessed October 28, 2011].

Corcoran, P.M., Bigioi, P. & Nanu, F., 2013. Half-face detector for enhanced performance of flash-eye filter. In 2013 IEEE International Conference on Consumer Electronics (ICCE). IEEE, pp, 252—253. Available at: http://ieeexplore.ieee.org/articleDetails.jsp?arnumber=6486882 [Accessed October 7, 2013].

Dainty, C., 2012. Film Photography Is Dead: Long Live Film: What can 15 Digital Photography Learn from the Film Era? IEEE Consumer Electronics Magazine, 1(1), pp.61—64. Available at: http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=610 7480 [Accessed July 29, 2013].

Tresadern, P.A., Ionita, M.C. & Cootes, T.F., 2011. Real-Time Facial 20 Feature Tracking on a Mobile Device. International Journal of Computer Vision, pp.1-10. Available at: http://www.springerlink.com/content/a0h707650870942h/ [Accessed December 1, 2011].

Yang, M. et al., 2010. AdaBoost-based face detection for embedded 25 systems. Computer Vision and Image Understanding, 114(11), pp.1116—1125. Available at: http://dx.doi.Org/10.1016/j.cviu.2010.03.010 [Accessed July 21, 2011].

Claims (5)

1. A peripheral for a handheld imaging device comprising a lens and image sensor for acquiring digital images, further comprising: an image signal processor configured to perform processing on a Bayer image obtained from said sensor; an interface module configured to convert a MIPI data stream into a USB-3 data stream; and further configured to receive commands from the handheld imaging device including at least a command to acquire a next image; and control means within the handheld imaging device to disable its internal imaging subsystem; and communication means between handheld imaging device and peripheral to enable the exchange of data and command codes; wherein the handheld imaging device disables its internal imaging subsystem, issues a command to the peripheral to acquire a next image and transmit said image via the communication means.

2. The peripheral of claim 1 including: means for adjusting one or more manual image acquisition parameters and associated camera settings prior to issuing a command to acquire a next image.

3. The peripheral of claim 1, further comprising: a data store configured to store at least a portion of a Bayer image obtained from the image sensor together with associated metadata, said metadata including at least image acquisition parameters associated with a Bayer image included in the MIPI data stream.

4. The peripheral of claim 1, further comprising: means for receiving of the acquired image from the peripheral by the handheld imaging device, wherein said device performs at least one of analyzing, processing or displaying said acquired image ,

5. The peripheral of claim 1 wherein: the communication means incorporates a conversion means from MIPI to USB-3 physical interface.