JP7613052B2 - IMAGING APPARATUS, DISTRIBUTION SYSTEM, DISTRIBUTION METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to an imaging device, a distribution system, a distribution method, and a program .

In recent years, technology that allows images captured using a camera to be streamed using a distribution service on the Internet has become widespread. Systems are also known that allow streaming distribution by connecting a camera directly to the Internet without going through a terminal such as a PC or smartphone.

Furthermore, in this type of streaming distribution, a technique is known that simultaneously distributes images from multiple streams of different image quality so that the viewing terminal can select the stream to receive. In this case, the processing power of the distribution device to encode the image data to be distributed is important. For example, Patent Document 1 discloses a method of encoding video data by allocating processing to a software encoder and a hardware encoder, and then synthesizing the encoded data into a single stream.

However, conventional methods have the problem that in devices with limited number of hardware encoders, encoding processes for each stream cannot be performed in parallel, making it difficult to simultaneously distribute image data from multiple streams.

In order to solve the above-mentioned problems, the invention of claim 1 is an imaging device that distributes image data of multiple streams, and is equipped with an acquisition means for acquiring image data captured by the imaging means, at least one hardware encoder and a software encoder, a decision means for deciding an encoding method for the acquired captured image data in accordance with encoding parameters for each stream, a generation means for generating streaming image data for each stream based on the determined encoding method, using the hardware encoder for a stream among the multiple streams that corresponds to the encoding parameter with a large processing load, and using the software encoder for the other streams, and a distribution means for distributing the generated multiple streaming image data.

According to the present invention, it is possible to simultaneously deliver multiple streams of image data using an imaging device with a limited number of hardware encoders.

FIG. 1 is a diagram illustrating an example of a system configuration of a distribution system according to a first embodiment. FIG. 2 is a diagram illustrating an example of an outline of processing of the distribution system according to the first embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of an imaging apparatus according to the first embodiment. FIG. 1 is a diagram illustrating an example of a hardware configuration of a communication terminal according to a first embodiment; FIG. 2 is a diagram illustrating an example of a functional configuration of a distribution system according to a first embodiment. FIG. 4 is a conceptual diagram illustrating an example of an encoding parameter management table according to the first embodiment. FIG. 4 is a sequence diagram showing an example of an image delivery process in the delivery system according to the first embodiment. 5 is a flowchart showing an example of an encoding process of image data according to the first embodiment. 13A and 13B are conceptual diagrams showing another example of the encoding parameter management table according to the first embodiment. 10 is a flowchart showing an example of an encoding process of image data according to a modified example of the first embodiment. 13 is a conceptual diagram for explaining an example of an encoding parameter management table when a target encoder is changed in the imaging device according to the modified example of the first embodiment. FIG. FIG. 11 is a diagram illustrating an example of a system configuration of a distribution system according to a second embodiment. FIG. 11 is a diagram illustrating an example of an outline of processing of a distribution system according to a second embodiment. FIG. 11 is a diagram illustrating an example of a functional configuration of a distribution system according to a second embodiment. FIG. 11 is a sequence diagram showing an example of an image delivery process in a delivery system according to a second embodiment.

Below, the mode for implementing the invention will be explained with reference to the drawings. Note that in the explanation of the drawings, the same elements are given the same reference numerals, and duplicate explanations will be omitted.

First embodiment
System Configuration Fig. 1 is a diagram showing an example of the system configuration of a distribution system according to the first embodiment. The distribution system 1 shown in Fig. 1 is a system that can perform streaming distribution of image data captured and acquired by an imaging device 10 to a communication terminal 50 used by a user.

As shown in FIG. 1, the distribution system 1 is composed of a photographing device 10 and a communication terminal 50. The photographing device 10 and the communication terminal 50 are communicatively connected via a communication network 5. The communication network 5 is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), etc. Note that the communication network 5 may include not only wired communication, but also wireless communication networks such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity (registered trademark)), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

The photographing device 10 is a digital camera that can photograph a subject and obtain a photographed image. The photographing device 10 is, for example, a special digital camera for obtaining a 360° panoramic image. The photographing device 10 may be a general digital camera (such as a single-lens reflex camera, a compact digital camera, a security camera, or a video (video) camera). The photographing device 10 streams image data related to the photographed image via a communication network 5 such as the Internet. The photographed image may be a video or a still image, or may be both a video and a still image. The photographed image may also include sound along with the image.

The communication terminal 50 is a terminal device such as a tablet terminal used by a user. The communication terminal 50 performs data communication with the image capture device 10 by accessing a communication network 5 such as the Internet. The communication terminal 50 can receive streaming image data distributed from the image capture device 10. Note that the communication terminal 50 is not limited to a tablet terminal, and may be a PC (Personal Computer), a smartphone, a mobile phone, a wearable terminal, a game console, a video conference (television conference) terminal, an IWB (Interactive White Board: an electronic whiteboard function that allows mutual communication), a telepresence robot, etc.

Overview Here, an overview of the processing of the distribution system according to the first embodiment will be described. Note that Fig. 2 is a simplified overview of the distribution system according to the first embodiment, and the details of the functions and the like realized by the distribution system 1 will be described later with reference to the drawings and the like.

Fig. 2 is a diagram showing an example of an outline of the processing of the distribution system according to the first embodiment. The distribution system 1 is a system that can distribute image data of multiple streams by assigning an encoder possessed by the imaging device 10 to each stream and processing the data in parallel when the image data is distributed in multiple streams.

First, the photographing device 10 acquires photographed image data to be the subject of image distribution. Next, the photographing device 10 prepares multiple streams with different image quality so that the communication terminal 50 can receive data with the desired image quality depending on, for example, the network bandwidth state of the communication network 5. The photographing device 10 determines the encoding method for the image data for each stream based on the number of streams to be streamed and the encoding parameters for each stream. The encoding parameters are information that indicate the image quality of the image data to be processed by the encoder. The photographing device 10 then performs encoding processing for each frame (e.g., frame X) of the photographed image data using the determined encoding method.

In the example of FIG. 2, the image capture device 10 performs encoding processing on stream 1 using a hardware encoder (HW), and performs encoding processing on streams 2 and 3 using a software encoder (SW). Frame X, which is one frame of captured image data, undergoes a predetermined image processing and is input in parallel to three encoders, which perform encoding processing in parallel according to the set encoding parameters.

Then, the imaging device 10 distributes the encoded streaming image data as independent streams to the communication terminal 50 via the communication network 5. In this case, for example, the streaming image data distributed in stream 1 is high-quality data, the streaming image data distributed in stream 2 is medium-quality data, and the streaming image data distributed in stream 3 is low-quality data.

When multiple streams of image data with different image quality are distributed simultaneously, the distribution device often uses a dedicated hardware encoder. However, for example, in a device with physical constraints such as device size or layout within the device, or constraints such as price, such as a typical digital camera, the number of hardware encoders that can be installed is limited, making it difficult to distribute multiple streams simultaneously. Therefore, when distributing image data using multiple streams with different image quality, the distribution system 1 assigns either a hardware encoder or a software encoder provided in the imaging device 10 to each stream according to the encoding parameters for each stream, and performs encoding processing. In this way, the distribution system 1 can distribute multiple streams of image data with different image quality using the imaging device 10, which has a limited number of hardware encoders.

Hardware Configuration Next, the hardware configuration of the photographing device 10 and the communication terminal 50 in the embodiment will be described with reference to Figures 3 and 4. Note that components may be added or deleted from the hardware configurations shown in Figures 3 and 4 as necessary.

○Hardware configuration of the imaging device○
First, the hardware configuration of the imaging device 10 will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the hardware configuration of the imaging device according to the first embodiment. In the following, the imaging device 10 is an all-sphere (omnidirectional) imaging device using two imaging elements, but the number of imaging elements may be two or more. In addition, the imaging device 10 does not necessarily have to be a device dedicated to omnidirectional imaging, and a normal digital camera, a smartphone, or the like may be provided with substantially the same functions as the imaging device 10 by attaching an omnidirectional imaging unit to it after the fact.

As shown in FIG. 3, the photographing device 10 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a hardware encoder 106, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an external storage I/F 116, an input/output I/F 117, a network I/F 118, and an acceleration/direction sensor 119.

Of these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b each having an angle of view of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b provided to correspond to each wide-angle lens. The imaging elements 103a and 103b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image captured by the wide-angle lenses 102a and 102b into image data of an electrical signal and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands and parameters required for the operation of the imaging elements are set.

The imaging elements 103a and 103b are each connected to the image processing unit 104 via a parallel I/F bus. The imaging elements 103a and 103b are also each connected to the imaging control unit 105 via a serial I/F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via a bus line 110. The bus line 110 is further connected to the ROM 112, the SRAM 113, the DRAM 114, the operation unit 115, the external storage I/F 116, the input/output I/F 117, the network I/F 118, and the acceleration/direction sensor 119.

The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I/F bus. The image processing unit 104 then performs a predetermined process on each piece of image data, and then synthesizes the image data to create equirectangular projection image data.

The imaging control unit 105 generally sets commands and the like in the registers of the imaging elements 103a and 103b using the I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the registers of the imaging elements 103a and 103b, and send it to the CPU 111.

In addition, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button on the operation unit 115 is pressed. The imaging device 10 may also have a preview display function on a display and a function corresponding to the display of still images and videos. In the case of videos, the image data is output from the imaging elements 103a and 103b continuously at a predetermined frame rate (frames/minute).

The imaging control unit 105 also functions as a synchronization control means that cooperates with the CPU 111 to synchronize the output timing of image data from the imaging elements 103a and 103b, as described below. Note that in this embodiment, the imaging device 10 is not provided with a display, but a display unit may be provided.

The hardware encoder 106 is an encoder that performs encoding (compression) using an encoding method such as H.264 or MPEG-4. The hardware encoder 106 is limited in the number of encoder processes that can be executed simultaneously.

The microphone 108 collects sound from the surrounding environment of the image capture device 10 and converts the collected sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I/F bus and performs a predetermined process on the sound data. The microphones 108 each constitute a sound collection means for collecting sound from the surrounding environment. Also, while FIG. 3 shows only the case where the microphone 108 is built into the image capture device 10, the microphone 108 may be externally attached to the image capture device 10. Furthermore, the number of microphones 108 is not limited to one, and there may be multiple microphones with a predetermined arrangement. Also, the microphone 108 may be an Ambisonics microphone.

The CPU 111 controls the overall operation of the image capture device 10 and executes necessary processing. The ROM 112 is a non-volatile memory that stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are volatile memories that are used as a work area for the CPU 111 and store programs executed by the CPU 111 and data in the middle of processing. In particular, the DRAM 114 stores, for example, image data in the middle of processing by the image processing unit 104 and data of a processed equirectangular projection image.

The operation unit 115 is a general term for operation buttons such as a shutter button. A user operates the operation unit 115 to input various shooting modes, shooting conditions, and the like. The external storage I/F 116 controls the reading and writing (storing) of data from and to an external storage such as a recording medium. The external storage may store encoded streaming image data as an image file (moving image file).

The input/output I/F 117 is an interface for connecting to external devices in order to input and output various data or information between the external devices. The input/output I/F 117 may be wireless or wired, and may be configured, for example, by a Universal Serial Bus (USB) I/F. The network I/F 118 is an interface for performing data communication via the communication network 5. The network I/F 118 is configured by a wired NIC (Network Interface Card), a wireless NIC, or a USB I/F.

The acceleration and orientation sensor 119 calculates the orientation of the image capture device 10 from the Earth's magnetism and outputs orientation information. The orientation information is an example of related information (metadata) that conforms to the Exchangeable image file format (Exif), and is used for image processing such as image correction of the captured image. The related information includes data such as the capture date and time of the image, and the data size of the image data.

The image capturing device 10 is not limited to an all-sphere (omnidirectional) image capturing device, but may be an ordinary digital camera, a surveillance camera, a video camera, or other image capturing device. In this case, the image capturing unit 101 as the image capturing means is composed of a pair of lenses and an image capturing element.

○Hardware configuration of communication terminal○
Next, the hardware configuration of the communication terminal 50 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the hardware configuration of the communication terminal according to the embodiment. The communication terminal 50 includes a CPU 501, a ROM 502, a RAM 503, an input device 504, a display 505, an external storage I/F 506, an input/output I/F 507, and a network I/F 508.

The CPU 501 controls the overall operation of the communication terminal 50. The ROM 502 is a non-volatile memory that stores programs used to drive the CPU 501. The RAM 503 is a volatile memory used as a work area for the CPU 501. The input device 504 is a type of input means that accepts operational input by the user. The input device 504 is, for example, a keyboard, a mouse, a touch panel, a button, a dial, or a microphone for voice input.

The display 505 is a type of display means such as an LCD or organic EL display that displays various information such as a cursor, menu, window, character, icon, or image. The external storage I/F 506 controls the reading or writing (storing) of data to an external storage such as a recording medium. The input/output I/F 507 is an interface for connecting to an external device in order to input/output various data or information between various external devices. The network I/F 508 is an interface for performing data communication via the communication network 5. The communication terminal 50 also includes a bus line 510. The bus line 510 is an address bus or data bus, etc. for electrically connecting each component such as the CPU 501.

The above programs may be distributed by recording them on a computer-readable recording medium in the form of an installable or executable file. Examples of recording media include CD-Rs (Compact Disc Recordable), DVDs (Digital Versatile Disks), Blu-ray (registered trademark) discs, SD cards, and USB memories. The recording media may be provided domestically or internationally as a program product. For example, the imaging device 10 realizes the distribution method according to the present invention by executing the program according to the present invention.

Functional Configuration Next, the functional configuration of the distribution system according to the first embodiment will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a diagram showing an example of the functional configuration of the distribution system according to the embodiment. Fig. 5 shows those devices and terminals shown in Fig. 1 that are related to the processing or operation described later.

○Functional configuration of the imaging device○
First, the functional configuration of the photographing device 10 will be described with reference to Fig. 5. The photographing device 10 has a communication unit 11, a reception unit 12, an input/output unit 13, a judgment unit 14, a photographing processing unit 15, a parameter setting unit 16, an image processing unit 17, an encoding processing unit 18, and a storage/reading unit 19. Each of these units is a function or a means that is realized by operating any of the components shown in Fig. 3 according to an instruction from the CPU 111 according to a program loaded from the ROM 112 or SRAM 113 onto the DRAM 114. The photographing device 10 also has a storage unit 1000 constructed by the ROM 112, SRAM 113, or DRAM 114 shown in Fig. 3.

The communication unit 11 is realized by commands from the CPU 111 shown in FIG. 3 and the network I/F 118, and has a function of transmitting and receiving various data (or information) to and from the communication terminal 50 via the communication network 5. The communication unit 11 distributes, for example, image data captured and acquired by the image capture processing unit 15 to the communication terminal 50 via the communication network 5. The communication unit 11 is an example of a distribution means.

The reception unit 12 is realized by commands from the CPU 111 shown in FIG. 3 and input means such as the operation unit 115, and is a function that receives various inputs from the user. The judgment unit 14 is realized by commands from the CPU 111 shown in FIG. 3 and is a function that makes various judgments. The input/output unit 13 is realized by commands from the CPU 111 shown in FIG. 3 and the input/output I/F 117, and is a function that inputs and outputs various data or information between external devices. The judgment unit 14 determines the encoding method of the captured image data, for example, according to the encoding parameters for each stream. Specifically, the judgment unit 14 determines whether the captured image data for each stream should be encoded using a hardware encoder or a software encoder. The judgment unit 14 is an example of a determination means.

The photographing processing unit 15 is realized by commands from the CPU 111 shown in FIG. 3, as well as the imaging unit 101, image processing unit 104, imaging control unit 105, microphone 108, and sound processing unit 109, and has the function of photographing a subject, a landscape image, etc., and acquiring photographed image data. The photographing processing unit 15 is an example of an acquisition means.

The parameter setting unit 16 is realized by commands from the CPU 111 shown in FIG. 3, and is a function that sets encoding parameters for each stream. The parameter setting unit 16 is an example of a setting means. The image processing unit 17 is realized by commands from the CPU 111 shown in FIG. 3, and is a function that performs various processes on the captured image data acquired by the capture processing unit 15.

The encoding processing unit 18 is realized by an instruction from the CPU 111 shown in FIG. 4, and is a function that performs encoding processing of the captured image data using a determined encoding method. The encoding processing unit 18 performs encoding processing using the determined encoding method, for example, to generate streaming image data for each stream. When the encoding method is a method using a hardware encoder, the encoding processing unit 18 performs encoding processing of the captured image data using the hardware encoder 106 shown in FIG. 3. On the other hand, when the encoding method is a method using a software encoder, the encoding processing unit 18 performs encoding processing of the captured image data using a software encoder that is operated by the CPU 111 shown in FIG. 3 executing dedicated software. The encoding processing unit 18 is an example of a generating means.

The memory/read unit 19 is realized by instructions from the CPU 111 shown in FIG. 3, and has the function of storing various data in the memory unit 1000 or reading various data from the memory unit 1000.

Encoding Parameter Management Table FIG. 6 is a conceptual diagram showing an example of an encoding parameter management table according to the first embodiment. An encoding parameter management DB 1001 configured by an encoding parameter management table as shown in FIG. 6 is constructed in the storage unit 1000. The encoding parameter management table stores and manages an encoding method for each stream using the encoding processing unit 18. The encoding parameter management table shown in FIG. 6 is an example of a case where one hardware encoder 106 as shown in FIG. 3 is provided in the imaging device 10. In addition, the number of encoding processes that the hardware encoder 106 can simultaneously execute in this case is one.

In the encoding parameter management table, encoding parameters and a target encoder indicating an encoder to be used in encoding are stored and managed in association with each stream that distributes image data. Among these, the encoding parameters indicate information indicating the image quality of the image data for each stream. The information indicating the image quality includes, for example, the resolution and frame rate of the image data. The encoding parameters may also include the bit rate of the image data. Furthermore, when the image data is moving image data, the encoding parameters are treated as information indicating the image quality as well as the sound quality of the audio data. Furthermore, the encoding parameters may also include the encoding method to be used in the encoding process. For example, for still images, the encoding methods include JPEG (Joint Photographic Experts Group) or its evolution (JPEG2000), or GIF (Graphics Interchange Format), and for moving images, the encoding methods include MPEG (Moving Picture Experts Group) or its evolution (H.264, H.265), and the like.

The target encoder is assigned, for example, to the hardware encoder 106 from streams 1 to 3 in descending order of encoding parameter processing load. In the encoding parameter management table, for example, the hardware encoder 106 is assigned to stream 1, which has the highest resolution, and the software encoder is assigned to streams 2 and 3.

○Functional configuration of communication terminal○
Next, the functional configuration of the communication terminal 50 will be described with reference to Fig. 5. The communication terminal 50 has a communication unit 51, a reception unit 52, a display control unit 53, a determination unit 54, and a storage/readout unit 59. Each of these units is a function or a means for performing a function that is realized when any of the components shown in Fig. 4 operates in response to an instruction from the CPU 501 according to a program deployed on the RAM 503. The communication terminal 50 also has a storage unit 5000 constructed by the ROM 502 or an external storage shown in Fig. 4.

The communication unit 51 is realized by commands from the CPU 501 shown in FIG. 4 and the network I/F 508, and has the function of transmitting and receiving various data or information to and from the image capture device 10 via the communication network 5.

The reception unit 52 is realized by commands from the CPU 501 shown in FIG. 4 and the input device 504, and has a function of receiving various input operations from the user. The display control unit 53 is realized by commands from the CPU 501 shown in FIG. 3 and the display 505, and has a function of displaying various screens, such as an operation screen that receives input operations by the user, on the display 505. The judgment unit 54 is realized by commands from the CPU 501 shown in FIG. 4, and has a function of making various judgments. The storage/reading unit 59 is executed by commands from the CPU 501 shown in FIG. 4, and has a function of storing various data in the storage unit 5000 or reading various data from the storage unit 5000.

Processing or Operation of First Embodiment Next, processing or operation of the distribution system according to the first embodiment will be described with reference to Fig. 7 to Fig. 11. First, an overall flow of processing or operation in the distribution system 1 will be described with reference to Fig. 7. Fig. 7 is a sequence diagram showing an example of image distribution processing in the distribution system according to the first embodiment.

First, the reception unit 52 of the communication terminal 50 receives a request to start streaming image distribution by receiving a predetermined input operation of the user on the input device 504 (step S11). The communication unit 51 transmits an image distribution request indicating a request for streaming image distribution to the imaging device 10 (step S12). As a result, the communication unit 11 of the imaging device 10 receives the image distribution request transmitted from the communication terminal 50. Note that the image distribution request transmitted from the communication terminal 50 may include information on the number of streams requesting image distribution and the encoding parameters for each stream. In this case, the information on the number of streams requesting image distribution and the encoding parameters for each stream included in the image distribution request is received by the reception unit 52 by the user inputting the input device 504. Also, the imaging device 10 may automatically start processing for data distribution, triggered by a user operation on the operation unit 115 or by turning on the power of the imaging device 10.

Next, the image capture processing unit 15 of the image capture device 10 executes image capture processing of the subject and acquires captured image data (step S13). Then, the image capture device 10 executes encoding processing of the captured image data acquired by the image capture processing unit 15 (step S14).

Here, the encoding process of the captured image data by the image capture device 10 will be described in detail with reference to FIG. 8. FIG. 8 is a flowchart showing an example of the encoding process of the image data according to the first embodiment.

First, the parameter setting unit 16 sets the number of streams to be streamed and the encoding parameters for each stream (step S101). Specifically, the parameter setting unit 16 sets the number of streams and the encoding parameters for each stream in the encoding parameter management DB 1001 (see FIG. 6) based on information included in the image distribution request received by the communication unit 11 or information input from an external device to the input/output unit 13. Note that the parameter setting unit 16 may set information stored in advance for streaming distribution as the encoding parameters. In the example shown in FIG. 6, the parameter setting unit 16 sets three streams and stores the encoding parameter information for each stream in the encoding parameter management DB 1001.

Next, the determination unit 14 determines the number of processes that can be simultaneously executed by the hardware encoder 106 (step S102). The determination unit 14 determines the number of processes that can be simultaneously executed by the hardware encoder 106 based on, for example, the processing capacity of the hardware encoder 106 or the values of the encoding parameters set in step S101. In the case of the encoding parameter management table shown in FIG. 6, the determination unit 14 determines that the number of processes that can be simultaneously executed is one.

Next, the determination unit 14 determines a target encoder for encoding each stream as the encoding method for the captured image data for each stream set in step S101 (step S103). Specifically, the determination unit 14 uses the encoding parameters set in step S101 to determine whether to use the hardware encoder 106 or the software encoder for encoding each stream. For example, the determination unit 14 determines the target encoder so that the hardware encoder 106 is assigned to a stream with a large processing load. The number of streams to which the hardware encoder 106 can be assigned corresponds to the number of streams that can be simultaneously processed determined in step S102.

The determination unit 14 assigns the hardware encoder 106 to high-resolution streams that have a large processing load, and assigns a software encoder to other streams, depending on the value of the image data resolution, which is one of the encoding parameters. When using the encoding parameter management table shown in FIG. 6, the determination unit 14 assigns the hardware encoder 106 to stream 1, which has the highest resolution, and assigns software encoders to streams 2 and 3.

The image capture processing unit 15 captures one frame of captured image data captured by the imaging unit 101, which is an example of an imaging means (step S104). The image processing unit 17 then executes image processing of the captured captured image data (step S105). The encoding processing unit 18 then executes encoding processing using the target encoder determined in step S103 on the image data processed in step S105, based on the encoding parameters set for each stream (step S106). The encoding processing unit 18 then executes encoding processing using the target encoder for each stream, and generates streaming image data for each stream. The encoding processing unit 18 processes the encoding processing using the target encoder for multiple streams and the generation of streaming image data in parallel using a hardware encoder and a software encoder.

Then, the communication unit 11 distributes the streaming image data, which is the captured image data encoded in step S106, to the communication terminal 50 (step S107). Note that the streaming image data may be distributed to the communication terminal 50 directly using the input/output unit 13, without going through the communication network 5.

Then, when the image capture device 10 has finished distributing all frames of captured image data acquired by the image capture processing unit 15 (YES in step S108), the image capture device 10 ends the process. On the other hand, when there are frames of captured image data acquired by the image capture processing unit 15 that have not been distributed (NO in step S108), the image capture device 10 repeats the process from step S104 to continue distributing the streaming image data.

As a result, when the imaging device 10 distributes multiple streams of image data with different image quality, it can determine the appropriate encoding method (whether to use a hardware or software encoder) depending on the image quality of the image data for each stream.

Returning to FIG. 7, as shown in step S107, the communication unit 11 of the imaging device 10 transmits (distributes) the multiple streaming image data generated by the encoding process in step S14 to the communication terminal 50 via the communication network 5 (step S15).

Next, the communication terminal 50 selects the data to be received from the multiple streaming image data distributed from the imaging device 10 (step S16). Specifically, the reception unit 52 of the communication terminal 50 accepts the selection of the streaming image data to be received by a predetermined input operation of the user on the input device 504. The judgment unit 54 of the communication terminal 50 selects the streaming image data to be received according to the network bandwidth information from the communication unit 51. In this case, the communication unit 11 monitors the communication state of the communication network 5, for example, based on the presence or absence of communication delays. The judgment unit 54 selects the streaming image data corresponding to the current communication state of the communication network 5. Then, the display control unit 53 of the communication terminal 50 displays the image related to the streaming image data selected and received in step S16 on the display 505 (step S17).

Here, another example of the encoding method of the captured image data determined by the above-mentioned determination unit 14 will be described. First, the encoding parameter management table shown in FIG. 9(A) is an example in which, unlike the example in FIG. 6, the number of encoding processes that can be simultaneously executed in one hardware encoder 106 as shown in FIG. 3 is two. In the encoding parameter management table shown in FIG. 9(A), for example, a hardware encoder is assigned to stream 1 with the highest resolution and stream 2 with the second highest resolution, and a software encoder is assigned to stream 3. In this way, the imaging device 10 can assign hardware encoders to the encoding processes of multiple streams depending on the number of hardware encoders 106 or the number of processes that can be simultaneously executed in one hardware encoder 106.

The encoding parameter management table shown in FIG. 9(B) differs from the example in FIG. 6 in that, among the encoding parameters, the frame rate differs for each stream. In the encoding parameter management table shown in FIG. 9(B), for example, a hardware encoder is assigned to stream 1, which has the highest frame rate, and a software encoder is assigned to streams 2 and 3. In this way, the imaging device 10 can determine the target encoder to be used for encoding processing for each stream by determining the processing load of the encoder using not only the resolution of the captured image data as an encoding parameter, but also the frame rate of the image data.

The type of encoding parameters used to determine the target encoder (encoding method) for each stream can be set arbitrarily. For example, the determination unit 54 may determine the target encoder by prioritizing the resolution value among multiple encoding parameters, or may determine the target encoder by prioritizing the frame rate value.

●Effects of the first embodiment Therefore, the distribution system according to the first embodiment can distribute image data of multiple streams simultaneously by generating streaming image data using an encoding method determined according to encoding parameters for each stream with different image quality.

Modification of the First Embodiment Next, a distribution system according to a modification of the first embodiment will be described with reference to Figures 10 and 11. The same configurations and functions as those of the above-mentioned embodiment are given the same reference numerals, and the description thereof will be omitted. The distribution system 1a according to the modification of the first embodiment is a system that can change the encoding parameters of image data while streaming image data is being distributed from the imaging device 10 by the above-mentioned distribution system 1.

Figure 10 is a flow chart showing an example of image data encoding processing according to a modified example of the first embodiment. Note that the processing from step S201 to step S208 shown in Figure 10 is similar to the processing from step S101 to step S108 shown in Figure 8, and therefore a description thereof will be omitted. If the image capture device 10 is continuing to distribute streaming image data (NO in step S208), the processing proceeds to step S209. The parameter setting unit 16 determines whether the encoding parameters have been changed (step S209). The parameter setting unit 16 determines whether the encoding parameters have been changed, and changes the values of the encoding parameters.

If the encoding parameters have been changed (YES in step S209), the parameter setting unit 16 of the imaging device 10 moves the process to step S210. Specifically, if a parameter change request is received from the communication terminal 50 by the communication unit 11 or the input/output unit 13, or if a parameter change request is received by the reception unit 12 through an input operation by the distributor, the parameter setting unit 16 changes the value of the encoding parameter stored in the encoding parameter management DB 1001. In the example shown in FIG. 11, the parameter setting unit 16 changes the frame rate of stream 2 from "10 fps" to "30 fps" among the encoding parameters in which the frame rate differs for each stream. On the other hand, if the parameter setting unit 16 does not change the encoding parameters (NO in step S209), the process from step S204 is repeated.

Next, the determination unit 14 changes the encoding method for each stream according to the changed encoding parameters, and changes the target encoder to be used for the encoding process (step S210). Specifically, in the example shown in FIG. 11, when the frame rate of stream 2 is changed, the determination unit 14 recalculates the encoder allocation, and changes the target encoder of stream 2, which has the highest frame rate and the largest processing load, from a software encoder to a hardware encoder, and changes the target encoder of stream 1, which has a frame rate lower than that of stream 2, from a hardware encoder to a software encoder.

Then, the imaging device 10 repeats the process from step S204 using the changed target encoder and continues distributing the streaming image data. On the other hand, if the encoding parameters have not been changed, the parameter setting unit 16 continues distributing the streaming image data encoded using the target encoder determined in step S203.

As a result, even if the encoding parameters are changed during the distribution of image data, the distribution system 1a can recalculate the encoder allocation and automatically change the encoding method, so that distribution can continue with the optimal encoder allocation.

Second embodiment
Next, the configuration of a distribution system according to the second embodiment will be described with reference to Figures 12 to 15. Note that the same configurations and functions as those of the first embodiment are given the same reference numerals, and the description thereof will be omitted. The distribution system according to the second embodiment is a system that can distribute streaming image data generated by an image capture device 10 to a communication terminal 50 via a distribution control device 90.

System Configuration Fig. 12 is a diagram showing an example of a system configuration of a distribution system according to the second embodiment. The distribution system 2 shown in Fig. 12 is obtained by adding a distribution control device 90 to the configuration according to the first embodiment. The distribution control device 90 can communicate with the image capture device 10 and the communication terminal 50 via the communication network 5.

The distribution control device 90 is a server computer that controls the distribution of streaming image data transmitted from the imaging device 10. The distribution control device 90 may be constructed by a single computer, or may be constructed by multiple computers to which each section (function or means) is divided and arbitrarily assigned. In addition, the hardware configuration of the distribution control device 90 is the same as that of the communication terminal 50 shown in FIG. 4 or a partially modified hardware configuration, so a description thereof will be omitted.

13 is a diagram showing an example of an outline of the process of the distribution system according to the second embodiment. Note that the process of generating streaming image data for each stream by the imaging device 10 is similar to the process described in the first embodiment.

The imaging device 10 transmits the streaming image data for each generated stream to the distribution control device 90 via the communication network 5. The distribution control device 90 stores the received multiple streaming data in the storage unit 9000. Then, in response to a request from the communication terminal 50, the distribution control device 90 distributes (transmits) the streaming image data transmitted from the imaging device 10 and stored in the storage unit 9000 to the communication terminal 50.

As shown in FIG. 13, the distribution control device 90 can simultaneously distribute streaming image data to multiple communication terminals 50 (communication terminal 50A and communication terminal 50B). The distribution control device 90 can also distribute streaming image data corresponding to a stream in response to a request from a communication terminal 50, and can distribute streaming image data of a different stream to each communication terminal 50. As a result, by using the distribution control device 90, the distribution system 2 can distribute streaming image data sent from the imaging device 10 in a stream appropriate for each communication terminal 50 depending on the network bandwidth status of each communication terminal 50, etc.

Functional Configuration Next, the functional configuration of the distribution system according to the second embodiment will be described. Fig. 14 is a diagram showing an example of the functional configuration of the distribution system according to the second embodiment. Note that the functional configurations of the imaging device 10 and the communication terminal 50 (communication terminals 50A and 50B) are the same as those shown in Fig. 5, and therefore will not be described.

The distribution control device 90 has a communication unit 91 and a storage/readout unit 99. Each of these units is a function or a means for performing a function that is realized when any of the components shown in FIG. 4 operates in response to an instruction from the CPU 501 according to a program deployed on the RAM 503. The distribution control device 90 also has a storage unit 9000 constructed using the ROM 502 shown in FIG. 4 or an external storage.

The communication unit 91 is realized by commands from the CPU 501 shown in FIG. 4 and the network I/F 508, and has a function of transmitting and receiving various data or information between the image capture device 10 and the communication terminal 50 via the communication network 5. The storage/reading unit 99 is executed by commands from the CPU 501 shown in FIG. 4, and has a function of storing various data in the storage unit 9000 or reading various data from the storage unit 9000. The storage unit 9000 stores streaming image data transmitted from the image capture device 10.

Processing or Operation of the Second Embodiment Next, processing or operation of the delivery system according to the second embodiment will be described with reference to Fig. 15. Fig. 15 is a sequence diagram showing an example of image delivery processing in the delivery system according to the second embodiment.

First, the communication unit 91 of the distribution control device 90 transmits an image acquisition request to the imaging device 10, indicating a request to acquire streaming images (step S51). As a result, the communication unit 11 of the imaging device 10 receives the image acquisition request transmitted from the distribution control device 90. Note that the image acquisition request transmitted from the distribution control device 90 may include information on the number of streams for which image acquisition is requested and the encoding parameters for each stream. In this case, the distribution control device 90 may, for example, read and transmit information previously stored in the storage unit 9000, or may transmit information received by input from the user to the distribution control device 90. In addition, the imaging device 10 may automatically start processing for image acquisition, triggered by a user operation on the operation unit 115 or by the power-on of the imaging device 10.

Next, the image capture processing unit 15 of the image capture device 10 performs image capture processing (step S52). Then, the image capture device 10 executes encoding processing on the captured image data acquired by the image capture processing unit 15 (step S53). Note that the encoding processing in step S53 is similar to the content shown in FIG. 8, so a description thereof will be omitted.

Next, the communication unit 11 of the imaging device 10 transmits (distributes) the multiple streaming data generated and encoded in step S14 to the distribution control device 90 via the communication network 5 (step S54). As a result, the communication unit 91 of the distribution control device 90 receives the multiple streaming image data transmitted from the imaging device 10. Then, the storage/reading unit 99 of the distribution control device 90 stores the streaming image data received by the communication unit 91 in the storage unit 9000 (step S55). In this way, the distribution control device 90 temporarily stores the streaming image data transmitted from the imaging device 10.

Next, in response to the user request accepted by the acceptance unit 52, the communication unit 51A of the communication terminal 50A transmits an image distribution request indicating a request for distribution of streaming images to the distribution control device 90 (step S56). As a result, the communication unit 91 of the distribution control device 90 receives the image distribution request transmitted from the communication terminal 50A. Next, the communication unit 91 of the distribution control device 90 transmits (distributes) the multiple streaming image data stored in the storage unit 9000 to the communication terminal 50A (step S57).

The communication terminal 50A selects the data to receive from the multiple streaming image data distributed from the distribution control device 90 (step S58). Specifically, the reception unit 52A of the communication terminal 50A accepts the selection of the streaming image data to be received by a predetermined input operation by the user on the input device 504. The judgment unit 54A of the communication terminal 50A also selects the streaming image data to be received according to the network bandwidth information from the communication unit 51A. Then, the display control unit 53A of the communication terminal 50A displays an image related to the streaming image data selected and received in step S58 on the display 505 (step S59).

Meanwhile, in response to the user request accepted by the acceptance unit 52B, the communication unit 51B of the communication terminal 50B transmits an image distribution request indicating a request for distribution of streaming images to the distribution control device 90 (step S60). As a result, the communication unit 91 of the distribution control device 90 receives the image distribution request transmitted from the communication terminal 50B. Next, the communication unit 91 of the distribution control device 90 transmits (distributes) the multiple streaming image data stored in the storage unit 9000 to the communication terminal 50B (step S61).

Similar to the process of step S58, the communication terminal 50B selects data to be received from the multiple streaming image data distributed from the distribution control device 90 (step S62). Then, the display control unit 53B of the communication terminal 50B displays an image related to the streaming image data selected and received in step S62 on the display 505 (step S63). Note that the order of the processes of steps S56 to S59 and steps S60 to S63 may be reversed, or may be performed in parallel.

●Effects of the second embodiment Therefore, by using the distribution control device 90, the distribution system of the second embodiment can distribute streaming image data transmitted from the shooting device 10 in a stream appropriate for each communication terminal 50, depending on the network bandwidth status of each communication terminal 50, etc.

●Summary●
As described above, the imaging device according to one embodiment of the present invention is an imaging device 10 that distributes image data of multiple streams, and includes an imaging processing unit 15 (an example of an acquisition unit) that acquires image data captured by an imaging unit 101 (an example of an imaging unit), a judgment unit 14 (an example of a determination unit) that determines an encoding method for the acquired image data in accordance with encoding parameters for each stream, an encoding processing unit 18 (an example of a generation unit) that generates streaming image data for each stream using the determined encoding method, and a communication unit 11 (an example of a distribution unit) that distributes the generated multiple streaming image data. As a result, the imaging device 10 can simultaneously distribute image data of multiple streams by performing encoding processing using an appropriate encoding method according to image quality.

The imaging device according to one embodiment of the present invention includes at least one hardware encoder 106 and a software encoder. In the imaging device 10, the encoding processing unit 18 (an example of a generating means) generates streaming image data using the hardware encoder 106 for a stream corresponding to encoding parameters of high resolution or high frame rate, which impose a large processing load. This allows the imaging device 10 to simultaneously distribute multiple streams of image data even when there are restrictions on the number of hardware encoders.

Furthermore, in the imaging device according to one embodiment of the present invention, the encoding processing unit 18 (an example of a generating means) generates multiple streaming image data in parallel using the hardware encoder 106 and the software encoder. This allows the imaging device 10, which has a limited number of hardware encoders, to simultaneously distribute multiple streams of image data.

The imaging device according to one embodiment of the present invention also includes a parameter setting unit 16 (an example of a setting means) that sets encoding parameters for each stream. In the imaging device 10, the judgment unit 14 (an example of a decision means) changes the determined encoding method when the set encoding parameters are changed. As a result, even if the encoding parameters are changed during the distribution of streaming image data, the imaging device 10 can recalculate the encoder allocation and automatically change to the encoding method, so that streaming distribution can be continued with the optimal encoder allocation.

●Additional Information●
Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

The various tables in the above-described embodiments may be generated by the learning effect of machine learning, and by classifying the data of each associated item by machine learning, tables may not be used. Here, machine learning is a technology for enabling a computer to acquire human-like learning capabilities, and refers to a technology in which a computer autonomously generates algorithms required for judgments such as data identification from learning data that is previously loaded, and applies these to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods, and any learning method for machine learning may be used.

So far, we have described a photographing device, a distribution system, a distribution method, and a recording medium according to one embodiment of the present invention, but the present invention is not limited to the above-mentioned embodiment, and can be modified within the scope of what a person skilled in the art can imagine, such as adding, changing, or deleting other embodiments, and any aspect is within the scope of the present invention as long as it achieves the functions and effects of the present invention.

1 Distribution system 5 Communication network 10 Photographing device 11 Communication unit (an example of a distribution means)
14 Judgment unit (an example of a decision means)
15 Shooting processing unit (an example of an acquisition means)
16 Parameter setting unit (an example of a setting means)
18 Encoding processing unit (an example of a generating means)
50 Communication terminal 90 Distribution control device 101 Imaging unit (an example of an imaging means)

JP 2011-77564 A

Claims (9)

1. An image capturing device that delivers a plurality of streams of image data,
An acquisition means for acquiring photographed image data photographed by the photographing means;
At least one of a hardware encoder and a software encoder;
a determination means for determining an encoding method for the acquired captured image data in accordance with an encoding parameter for each stream;
a generating means for generating streaming image data for each of the plurality of streams based on the determined encoding method, by using the hardware encoder for a stream corresponding to the encoding parameter having a large processing load among the plurality of streams and by using the software encoder for the other streams ;
a distribution means for distributing the generated plurality of streaming image data;
An imaging device comprising: the encoding parameter is a resolution of the streaming image data;
2. The image capturing apparatus according to claim 1 , wherein the generating means generates the streaming image data for the high resolution stream by using the hardware encoder. the encoding parameter is a frame rate of the streaming image data;
3. The image capturing apparatus according to claim 1 , wherein the generating means generates the streaming image data using the hardware encoder for the stream with the high frame rate. 4. The image capturing apparatus according to claim 1 , wherein the generating means generates a plurality of streaming image data in parallel using the hardware encoder and the software encoder. A setting means for setting an encoding parameter for each of the streams,
5. The photographing apparatus according to claim 1, wherein the determination means determines an encoding method for the acquired photographic image data in accordance with a set encoding parameter. 6. The imaging apparatus according to claim 5 , wherein the determining means changes the determined encoding method when a set encoding parameter is changed. A distribution system for distributing a plurality of streams of image data, comprising:
An acquisition means for acquiring photographed image data photographed by the photographing means;
At least one of a hardware encoder and a software encoder;
a determination means for determining an encoding method for the acquired captured image data in accordance with an encoding parameter for each stream;
a generating means for generating streaming image data for each of the plurality of streams based on the determined encoding method, by using the hardware encoder for a stream corresponding to the encoding parameter having a large processing load among the plurality of streams and by using the software encoder for the other streams ;
An imaging device comprising:
a distribution means for distributing the plurality of streaming image data generated by the imaging device ;
A distribution system comprising: A distribution method executed by an image capture device that distributes a plurality of streams of image data, comprising:
An acquisition step of acquiring photographed image data photographed by a photographing means;
a determining step of determining an encoding method for the acquired captured image data in accordance with an encoding parameter for each stream;
a generating step of generating streaming image data for each of the plurality of streams based on the determined encoding method , by using at least one hardware encoder provided in the imaging device for a stream corresponding to the encoding parameter having a large processing load among the plurality of streams, and by using a software encoder provided in the imaging device for the other streams ;
a distribution step of distributing the generated plurality of streaming image data;
The delivery method to be performed. A computer of an imaging device that delivers multiple streams of image data,
An acquisition step of acquiring photographed image data photographed by a photographing means;
a determining step of determining an encoding method for the acquired captured image data in accordance with an encoding parameter for each stream;
a generating step of generating streaming image data for each of the plurality of streams based on the determined encoding method , by using at least one hardware encoder provided in the imaging device for a stream corresponding to the encoding parameter having a large processing load among the plurality of streams, and by using a software encoder provided in the imaging device for the other streams ;
a distribution step of distributing the generated plurality of streaming image data;
A program for executing.