JP7705273B2 - Imaging system and control method thereof - Google Patents

Description

The present invention relates to technology for remote control of imaging devices such as cameras.

There is a method in which an information processing device such as a smartphone, tablet device, or PC communicates with an imaging device via wired or wireless communication, and the imaging device is remotely controlled from the information processing device.

The imaging device is remotely controlled by the information processing device by the imaging device receiving a control signal from the information processing device via a communication means.

In addition, there is a configuration in which the imaging device can transmit to the information processing device a live view (hereinafter, LV) of the imaging device and an on-screen display (OSD) containing control information for the imaging device to be displayed on the display of the imaging device. In this configuration, the information processing device can display the LV and OSD of the imaging device received from the imaging device on the display of the information processing device. By displaying the LV and OSD of the imaging device on the display of the information processing device, the information processing device can remotely control the imaging device while checking both images.

On the other hand, when the LV and a display device such as the display of the imaging device have different attitudes, there is a technology that rotates the LV by 90 degrees and displays it on the display of the imaging device, thereby avoiding the LV being displayed sideways when viewed by the display of the imaging device (Patent Document 1).

Furthermore, when the LV is displayed on the display of an information processing device or the like, the LV and the display of the information processing device or the like may be in different positions. In the above-mentioned state, if it becomes necessary to rotate the LV in increments other than 90 degrees, the LV cannot be displayed on the display at the correct angle. There is a technology that solves this problem by attaching angle information at the time of shooting to the LV and rotating and displaying the LV in the opposite direction in a way that cancels out the angle information on the display (Patent Document 2).

Japanese Unexamined Patent Publication No. 58-182976 JP 2020-96349 A

However, in the above conventional example, when the LV and OSD are transmitted from the imaging device and the LV and OSD are displayed on the information processing device, if the display orientation of the OSD differs from the orientation of the LV, the OSD may not be viewable at a normal angle on the display of the information processing device using only the orientation information of the LV.

In order to solve this problem, for example, an imaging system according to the present invention has the following configuration.
An imaging system including a camera and a terminal that communicates with the camera and remotely controls the camera,
a transmission means for transmitting a live view and an OSD (On Screen Display) obtained by the camera and their respective attitude information from the camera to the terminal;
a display unit for displaying the live view, the OSD, and the respective attitude information on a screen of the terminal,
The display means is characterized in that, when displaying either a live view or an OSD, it displays it in an orientation based on the attitude information of the display object, and, when displaying both the live view and the OSD, it determines the orientations of the OSD and the live view based on the attitude information of the OSD and displays them in a superimposed manner.

According to the present invention, the OSD displayed on the display of a terminal such as an information processing device is in the correct display position, improving visibility when performing remote operation.

FIG. 1 is a block diagram showing the configuration of a camera to which an embodiment can be applied. FIG. 1 is a block diagram of a terminal to which the embodiment can be applied. FIG. 2 is a diagram showing an example of main communications in the embodiment. 5A and 5B are diagrams showing a sequence of LV and OSD transfer and display rotation setting in the embodiment. FIG. 4 is a process flow diagram of LV and OSD display setting according to the embodiment. 5A and 5B are diagrams showing examples of LV and OSD screen displays in the embodiment. 5A and 5B are diagrams showing examples of LV and OSD screen displays in the embodiment. 5A and 5B are diagrams showing examples of LV and OSD screen displays in the embodiment.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

Figure 1 is a block diagram of a camera 100, which is a member of the imaging system of this embodiment. Also, Figure 2 is a block diagram of a terminal 200, such as an information processing device, which is another member of the image processing system. The camera 100 in this embodiment is a still camera or a video camera, but is not limited to these. In short, any device having a camera function (imaging function) may be used, for example, a smartphone, a tablet, etc. Also, the terminal 200 is described as a personal computer in this embodiment, but is not limited to this. For example, the terminal 200 may be any device having a display function and various instruction input functions, for example, a mobile phone such as a smartphone, a so-called tablet terminal, a television, etc. may be used.

In FIG. 1, the camera 100 is composed of a CPU 101, a volatile memory 102, a non-volatile memory 103, a camera unit 104, a camera signal processing unit 105, a communication I/F 106, a connector/antenna 107, a recording medium I/F 108, a recording medium 109, an input I/F 110, an operation unit 111, an output I/F 112, a display 113, and an internal bus 120. Each unit is controlled by a program running on the CPU 101, and each unit connected to the internal bus 120 is capable of exchanging data with each other via the internal bus 120.

Volatile memory 102 is a memory in which programs, variables, temporary work data, etc. required by CPU 101 during operation are appropriately recorded, and is typically a DRAM. Non-volatile memory 103 is a memory in which various programs and parameters required for CPU 101 to operate are stored and held, and is typically a ROM or flash memory.

The CPU 101 controls each part of the camera 100 according to a program stored in the non-volatile memory 103 or the recording medium 109, using the volatile memory 102 as a work memory.

The camera unit 104 forms an optical image of a subject, converts the optical image into an analog electrical signal, and then converts it into a digital signal. Based on the control of the CPU 101, the camera signal processing unit 105 compresses and encodes the digital signal converted by the camera unit 104 at a predetermined bit rate and format, or decodes the compressed video encoding data.

The communication I/F 106 communicates with the terminal 200 via the connector/antenna 107 under the control of the CPU 101.

The recording medium I/F 108 connects a recording medium 109 such as a HDD or non-volatile memory, and reads data from the connected recording medium 109 and writes data to the recording medium 109 based on the control of the CPU 101. Note that the recording medium 109 to which the recording medium I/F 108 can be connected may be a removable non-volatile memory such as a memory card, for example, connected via a socket (not shown).

The input I/F 110 accepts user operations on the operation unit 111, generates control signals corresponding to the operations, and supplies them to the CPU 101. For example, the operation unit 111 has input devices for accepting user operations, such as a character information input device such as a zoom lever or a keyboard, and a pointing device such as a mouse or a touch panel. It also includes devices that can be operated remotely, such as an infrared remote control. Note that a touch panel is an input device that outputs coordinate information corresponding to the position touched on an input unit configured in a planar manner, for example. This allows the camera 100 to perform an operation corresponding to the user operation.

The output I/F 112 outputs a display signal for displaying on the display 113 based on display data such as a GUI generated by the CPU 101 according to a program or a live view (LV) composed of digital signals obtained from the camera unit 104. The LV can be displayed when a still image is in a standby state, when a video is in a standby state, or when a video is being recorded, and the captured subject image is displayed almost in real time. When a touch panel is used as the operation unit 111, the operation unit 111 and the display 113 can be configured as an integrated unit. For example, the touch panel is configured so that the light transmittance does not interfere with the display of the display 113, and is attached to the upper layer of the display surface of the display 113. Then, the input coordinates on the touch panel are associated with the display coordinates on the display 113. This makes it possible to configure a GUI that makes it seem as if the user can directly operate the screen displayed on the display 113.

The attitude sensor 114 detects the attitude (direction of gravity) of the camera 100. As a result, the CPU 101 can determine in what attitude the user is holding and posing the camera 100. In other words, the CPU 101 can determine whether the camera is being held for portrait photography or landscape photography. Note that a portrait refers to a state in which the vertical length of the image to be captured is longer than the horizontal length, and a landscape refers to a state in which the vertical length is shorter than the horizontal length.

In the embodiment, there are two types of OSD (On Screen Display) display: portrait and landscape. Displaying the OSD in portrait refers to a state in which the vertical length of a circumscribing rectangle containing a group of symbols such as characters and symbols that represent various shooting-related states and setting values is longer than its horizontal length. Conversely, displaying the OSD in landscape refers to a state in which the vertical length of a circumscribing rectangle containing a group of symbols such as characters and symbols is shorter than its horizontal length. Although there are many types of OSD to be displayed, the CPU 101 knows in advance the layout of the SD to be displayed. In other words, the CPU 101 also knows whether the OSD to be displayed will be displayed in portrait or landscape.

In FIG. 2, the terminal 200 is composed of a CPU 201, a volatile memory 202, a non-volatile memory 203, a communication I/F 204, a connector/antenna 205, a recording medium I/F 206, a recording medium 207, an input I/F 208, an operation unit 209, an output I/F 210, a display 211, and an internal bus 220. The components with reference numbers 201 to 211 basically have the same functions as the components with the same names in FIG. 1, so their explanations will be omitted.

The camera 100 communicates with the terminal 200 via the communication I/F 106 and the communication I/F 204, but the type of communication interface, including wired and wireless, is not important. The user of the terminal 200 looks at the live view and OSD displayed on the display 211 and performs various operations to remotely control the camera 100. The CPU 201 of the terminal 200 performs remote control by sending various commands corresponding to these operations to the camera 100, but since the focus of the present invention is on the processing related to the display of the live view and OSD sent from the camera 100, a description of remote control will be omitted.

The following describes the general operation of remote control in this embodiment using Figures 1, 2, and 3.

When a user inputs an instruction to execute remote control from the operation unit 111 of the camera 100, the CPU 101 of the camera 100 sets the communication I/F 106 to a communication-enabled state.

Furthermore, when the user operates the operation unit 209 (including the touch panel) of the terminal 200 to input a command to start a program such as an application browser required for communication connection processing and remote control, the CPU 201 of the terminal 200 controls the communication I/F 204 in accordance with the program stored in the non-volatile memory 203 or recording medium 207, starts communication with the camera 100, and performs connection processing.

When the connection process (establishment of communication) between the camera 100 and the terminal 200 is complete, the camera 100 and the terminal 200 start the processes required for remote control.

Figure 3 shows an example of a configuration in which, under remote control, the camera 100 transmits the LV 301 and OSD 302 stored in the volatile memory 102 to the terminal 200 under the control of the CPU 101.

Under the control of CPU 101, camera 100 can transmit LV301 (video data captured by camera unit 104) and OSD302, as well as their respective attitude information or display direction as additional information. Attitude information is information indicating the angle/screen direction relative to the display reference during shooting by camera 100, and the attitude information of LV301 is determined according to the attitude detected by attitude sensor 114, and the attitude information of OSD is determined by the type of OSD information to be displayed (or the type of item to be displayed). The display direction is an instruction for the rotation direction when LV301 and OSD302 are rotated and displayed on terminal 200. LV301 and OSD302 basically have the same attitude information and display direction, but when a specific operation such as menu display is performed on OSD302, OSD302 may have attitude information and display information different from LV301. Under the control of CPU 201, terminal 200 displays LV 301 and OSD 302 received from camera 100 in image display area 304 in a form in which OSD 302 is superimposed on LV 301. Under the control of CPU 201, image display area 304 can display images of LV 301 and OSD 302 similar to image display area 303 of camera 100.

Under the control of CPU 201, terminal 200 can rotate LV 301 and OSD 302 using the additional information of LV 301 and OSD 302 and display them in image display area 304. When OSD 302 is displayed, CPU 201 uses only the additional information of OSD 302 to rotate LV 301 and OSD 302. However, if image display area 304 is larger than a preset size, CPU 201 determines the display direction by rotating LV 301 and OSD 302 using the additional information of LV 301 and OSD 302, respectively. This ensures that characters, icons, etc. of OSD 302 displayed in image display area 304 are always displayed in the correct orientation.

Reference numeral 305 in the figure denotes a group of icons for remote control, and the user can operate these icons to send remote control instructions to the camera 100.

Reference number 306 is an icon that allows the user to select whether LV 301 is on or off. Normally, LV 301 is displayed, but depending on the processing capacity and communication load status of terminal 200, it is possible to turn off the LV display (hide the LV) by operating this icon, making it possible to prioritize operability.

Reference number 307 is an icon for selecting whether to turn OSD 302 on or off. Normally, OSD 302 is displayed, but depending on the processing capacity and communication load status of terminal 200, it is possible to turn off the OSD display by operating this icon, making it possible to prioritize operability.

Figures 4(a) to (c) show a sequence showing the rotation and display process of LV 301 and OSD 302 by camera 100 and terminal 200. This sequence is realized by CPU 101 of camera 100 controlling each part of camera 100 based on a program stored in non-volatile memory 103 or recording medium 109, and CPU 201 of terminal 200 controlling each part of terminal 200 based on a program stored in non-volatile memory 203 or recording medium 207.

Figure 4 (a) shows a sequence in which the display direction is set based on the attitude information of the LV 301 and OSD 302 under the control of the CPU 201 of the terminal 200, and the LV 301 and OSD 302 are rotated and displayed in the image display area 304.

Reference numeral 401 is a request from the terminal 200 to the camera 100 to send LV301 and OSD302, and attitude information of LV301 and OSD302. Reference numeral 402 indicates a response to the request 401 from the camera 100 to the terminal 200 (sending attitude information of LV301 and OSD302). This attitude information includes information indicating the gravity direction of LV301 and the gravity direction of OSD (or information indicating the up (or down) direction when displayed as a landscape). The attitude with respect to LV301 is determined by the attitude sensor 114. In addition, the attitude of OSD is determined according to the layout of the OSD to be displayed. Reference numeral 403 indicates a process in the terminal 200 to set the display direction based on the attitude information of LV301 and OSD302. Reference number 405 is the process of rotating the LV 301 and OSD 302 in the set display direction and displaying them in the image display area 304.

Figure 4 (b) shows a sequence in which the display direction based on the attitude information of the LV 301 and OSD 302 is set under the control of the CPU 101 of the camera 100, and the CPU 201 of the terminal 200 rotates and displays the LV 301 and OSD 302 in the image display area 304.

Reference numeral 405 denotes a request from the terminal 200 to the camera 100 to obtain the display direction of the LV 301 and OSD 302. Reference numeral 406 denotes processing in the camera 100 to set the display direction based on the attitude information of the LV 301 and OSD 302. Reference numeral 407 denotes the transmission of the LV 301 and OSD 302 from the camera 100 to the terminal 200, and the transmission of setting information related to their display directions. Reference numeral 408 denotes the rotation and display processing of the LV 301 and OSD 302 based on the information received by the terminal 200.

Figure 4 (c) shows a sequence in which, under the control of the CPU 201 of the terminal 200, the CPU 101 of the camera 100 sets the display direction based on the attitude information of the LV 301 and OSD 302, rotates the LV 301 and OSD 302, and then transmits them to the terminal 200. Upon receiving this, under the control of the CPU 201 of the terminal 200, the LV 301 and OSD 302 are displayed in the image display area 304.

Reference numeral 409 in the figure indicates a request to send LV 301 and OSD 302 from the terminal 200 to the camera 100. Reference numeral 410 indicates processing in which the CPU 101 of the camera 100 sets the display direction based on the attitude information of LV 301 and OSD 302. Reference numeral 411 indicates processing in which the CPU 101 of the camera 100 rotates LV 301 and OSD 302 according to the set direction. Reference numeral 412 indicates data transmission of LV 301 and OSD 302 from the camera 100 to the terminal 200. And reference numeral 413 indicates display processing of LV 301 and OSD 302 in the image display area 304 by the CPU 201 of the terminal 200.

Figure 5 shows the flow for setting the display direction based on S403, S406, S410, and the attitude information of SLV301 and OSD302 in Figures 4(a) to (c).

To simplify the explanation, the following description will be given assuming that the process in FIG. 4 is the process of S403, i.e., the process of the CPU 201 of the terminal 200. In other words, the CPU 201 of the terminal 200 receives the orientation information of the LV 301 and the OSD 302 from the camera 100, and sets the display direction of each based on the received information.

First, in S501, the CPU 201 determines the display mode of the terminal 200. The display mode follows the on (display)/off (non-display) setting of the icons 306 and 307. If both are set to "off (non-display)," no display processing is performed, so there are three possible cases to determine: {LV301 on, OSD302 off}, {LV301 off, OSD on}, and {LV301 on, OSD on}.

If the CPU 201 determines in S501 that the display mode of the terminal 200 is {LV301 on, OSD302 off}, the process proceeds to S502. In S502, the CPU 201 determines the orientation of the LV301. If the CPU 201 determines that the orientation of the LV301 is portrait, the process proceeds to S505, where it determines that the LV301 is to be displayed in portrait mode. Also, if the CPU 201 determines in S502 that the orientation of the LV301 is landscape, the CPU 201 proceeds to S506, where it determines that the LV301 is to be displayed in landscape mode.

If the CPU 201 determines in S501 that the display mode of the terminal 200 is {LV301 off, OSD302 on}, the process proceeds to S503. In S503, the CPU 201 determines the orientation of the OSD 302. If the CPU 201 determines that the orientation of the OSD 302 is portrait, the process proceeds to S507, where it determines that the OSD 302 is to be displayed in portrait mode. Also, if the CPU 201 determines in S503 that the orientation of the OSD 302 is landscape, the CPU 201 proceeds to S508, where it determines that the OSD 302 is to be displayed in landscape mode.

If the CPU 201 determines in S501 that the display mode of the terminal 200 is {LV301 on, OSD302 on}, the process proceeds to S504. In S504, the CPU 201 determines the orientation of the OSD 302. If the CPU 201 determines that the orientation of the OSD 302 is portrait, the process proceeds to S509, where it determines that both the LV 301 and the OSD 302 are to be displayed in portrait mode. If the CPU 201 determines in S504 that the orientation of the OSD 302 is landscape, the CPU 201 proceeds to S510, where it determines that both the LV 301 and the OSD 302 are to be displayed in landscape mode. In other words, the OSD orientation takes priority over the LV 301 orientation.

The above is the process indicated by reference number 403 in FIG. 4(a), but after this, the CPU 201 of the terminal 200 will display LV 301 (if on) and OSD 302 (if on) according to the determined display direction at reference number 404.

When applied to the processes of reference numerals 406 and S410 in FIG. 4, the subject that performs the processes is the CPU 101 of the camera 100. Therefore, when the CPU 201 of the terminal 200 sends an information acquisition request to the camera 101 at reference numerals 405 and 409, it also sends the current display mode (on/off status information for the LV and OSD). In particular, when applied to the process of 410, the CPU 101 performs a rotation process to match the LV 301 to the attitude of the OSD 302 (reference numeral 411).

Figure 6 (a) shows a case where only LV301 is displayed under the control of the CPU 201 of the terminal 200, and shows LV301 displayed in the image display area 304 after S506 in Figure 5 is executed. As shown in the figure, LV301 is displayed in landscape mode.

Figure 6 (b) shows a case where only OSD 302 is displayed under the control of the CPU 201 of the terminal 200, and shows OSD 302 displayed in the image display area 304 after step S508 in Figure 5 is executed. As shown in the figure, OSD 302 is displayed in landscape orientation.

Figure 6 (c) shows a case where both LV301 and OSD302 are displayed under the control of the CPU 201 of the terminal 200 (more precisely, OSD302 is displayed superimposed on LV301), and shows LV301 and OSD302 displayed in the image display area 304 after S510 in Figure 5 is executed. As shown in the figure, both LV301 and OSD302 are displayed in landscape orientation.

Figure 7 (a) shows a case where only LV301 is displayed under the control of the CPU 201 of the terminal 200, and shows LV301 displayed in the image display area 304 after S505 in Figure 5 is executed. As shown in the figure, LV301 is displayed in portrait mode.

Figure 7 (b) shows a case where only OSD 302 is displayed under the control of the CPU 201 of the terminal 200, and shows OSD 302 displayed in the image display area 304 after step S507 in Figure 5 is executed. As shown in the figure, OSD 302 is displayed in portrait mode.

Figure 7 (c) shows a case where LV301 and OSD302 are displayed under the control of CPU 201 of terminal 200, and shows LV301 and OSD302 displayed in image display area 304 after S509 in Figure 5 is executed. As shown in the figure, both LV301 and OSD302 are displayed in portrait mode.

Figure 8 (a) shows a case where LV301 and OSD302 are displayed under the control of CPU 201 of terminal 200, and shows LV301 and OSD302 displayed in image display area 304 after S510 in Figure 5 is executed. As shown in the figure, OSD302 is displayed in landscape orientation, and LV301 is also displayed in landscape orientation (rotated 90 degrees relative to the direction of gravity).

When the display 211 of the terminal 200 is large enough for the LV 301 and OSD 302 (for example, when the vertical and horizontal resolutions of the display area 304 are equal to or greater than the resolution of either side of the captured image), the LV 301 and OSD 302 may be displayed in the optimal orientation based on their respective supplementary information under the control of the CPU 201 of the terminal 200. Figure 8(b) shows this state, in other words, the LV 301 is displayed in portrait and the OSD 302 is displayed in landscape.

As described above, according to this embodiment, the OSD displayed on the remote control terminal (information processing device) is in an upright position, improving visibility when performing remote control.

Other Examples
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

100...camera, 101...CPU, 102...volatile memory, 103...non-volatile memory, 104...camera unit, 105...camera signal processing unit, 106...communication I/F, 107...connector/antenna, 108...recording medium I/F, 109...recording medium, 110...input I/F, 111...operation unit, 112...output I/F, 113...display, 114...orientation sensor, 120...internal bus, 200...terminal, 201...CPU, 202... volatile memory, 203... non-volatile memory, 204... communication I/F, 205... connector/antenna, 206... recording medium I/F, 207... recording medium, 208... input I/F, 209... operation unit, 210... output I/F, 211... display, 212... internal bus, 301... live view (LV), 302... on-screen display (OSD), 306... LV on/off icon, 307... OSD on/off icon

Claims (6)

An imaging system including a camera and a terminal that communicates with the camera and remotely controls the camera,
a transmission means for transmitting a live view and an OSD (On Screen Display) obtained by the camera and their respective attitude information from the camera to the terminal;
a display control means for displaying the live view, the OSD, and the respective attitude information on a screen of the terminal,
wherein when displaying either a live view or an OSD, the display control means displays the display in an orientation based on attitude information of the display object, and when displaying both the live view and the OSD, the display control means determines the orientations of the OSD and the live view based on the attitude information of the OSD and displays them in a superimposed manner. The imaging system according to claim 1, further comprising a selection means for selecting whether to display or hide the live view and whether to display or hide the OSD. The display control means
The imaging system according to claim 2, characterized in that, when the selection means selects to display only one of live view and OSD, the one selected for display is displayed in an orientation based on the corresponding attitude information, and, when the selection means selects to display both live view and OSD, the direction in which the live view is displayed is determined based on the attitude information of the OSD, and both the OSD and the live view are displayed. The terminal has the selection means,
The imaging system according to claim 2, characterized in that the terminal requests orientation information of live view and OSD from the camera, determines the orientation information of the live view and OSD based on information received as a response to the request from the camera and the selection made by the selection means, and determines the orientation of the OSD and live view based on the determination. the terminal has the selection means, and transmits state information related to a selection made by the selection means to the camera;
The imaging system according to claim 2 , wherein the camera determines the orientation of an OSD and a live view based on the state information. A method for controlling an imaging system including a camera and a terminal that communicates with the camera and remotely controls the camera, comprising:
a transmission step of transmitting a live view, an OSD (On Screen Display), and respective attitude information obtained by the camera from the camera to the terminal;
a display control step of displaying the live view, the OSD, and the respective attitude information on a screen of the terminal,
a display control step of displaying either a live view or an OSD in a direction based on attitude information of a display object, and a control step of displaying both the live view and the OSD in a direction based on attitude information of the OSD, the control step being characterized in that the display control step of the display control step determines the orientations of the OSD and the live view based on the attitude information of the OSD, and displays them in a superimposed manner.

Images