JP7129283B2 - Processing device, control method, and program - Google Patents

Description

The present invention relates to technology for operating a system using a plurality of devices.

A technique of installing a plurality of cameras at different positions, performing multi-viewpoint synchronous shooting, and generating virtual-viewpoint content using multi-viewpoint images obtained by the shooting has attracted attention (see Patent Document 1). According to this, for example, it is possible to view the highlight scenes of a soccer or basketball game, a concert, or the like from various angles. can give.

JP 2008-015756 A JP-A-9-139872

When photographing an event such as a sports game or a concert, it is necessary to install equipment such as a camera as a preparatory step. During the installation work, staff communicate with each other through voice communication using intercoms, etc., grasp the installation status of equipment, and confirm the work progress manually by checking checklists such as work manuals. is performed (see Patent Document 2). On the other hand, it is not easy to accurately grasp the details of the situation using only voice communication. Tasks can be difficult to complete. In addition, there is concern that human error such as mishearing may occur with only voice communication.

The present invention has been made in view of such problems, and provides a technique for accurately presenting information regarding work on a plurality of pieces of equipment.

A processing apparatus according to an aspect of the present invention relates to a task related to a plurality of pieces of equipment for making a system operating a plurality of pieces of equipment ready to perform a predetermined function, or related to the plurality of pieces of equipment for withdrawal of the system. setting means for setting a task; acquisition means for acquiring information including the connection status of each of the plurality of devices to the system from devices to which the plurality of devices are connected; and control means for causing a display means to display a screen regarding the progress of the tasks based on the tasks relating to the plurality of equipment and the information acquired by the acquisition means.

ADVANTAGE OF THE INVENTION According to this invention, the information regarding the operation|work about several equipment can be accurately shown.

1 is a diagram showing a configuration example of a multi-camera operation system; FIG. It is a figure which shows the functional structural example of an image processing system. It is a figure which shows the functional structural example of a camera adapter. FIG. 4 is a diagram showing an example of a workflow for operating the system; FIG. FIG. 7 is a diagram showing an example of the flow of processing when equipment is installed; FIG. 10 is a diagram showing an example of screen display on the camera operation terminal; It is a figure for demonstrating the example of a display of a correction plan. 3 is a diagram showing an example of the functional configuration of a camera operation terminal; FIG. FIG. 4 is a diagram showing an example of information held by a camera operation terminal; FIG. 11 is a diagram for explaining another display example of a correction plan; 4 is a diagram illustrating an example hardware configuration of a camera operation terminal; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the configurations described in the following embodiments are merely examples, and are not intended to limit the present invention to the described embodiments.

(System configuration)
In this embodiment, an example of a multi-camera operation system will be described in which a plurality of cameras are installed in facilities such as stadiums, concert halls, studios, and the like to take pictures. In the present embodiment, in particular, by visually collecting and notifying information in the installation (preparation for enabling the system to perform a predetermined function) and withdrawal of such a multi-camera operation system, the work can be performed. Allows you to set the progression appropriately. According to this, it is possible to suppress the occurrence of human errors such as mishearing by visual display compared to situation confirmation only by voice communication. At this time, by combining voice communication and visual information presentation according to the present embodiment, confirmation and the like can be performed more smoothly. In addition, since the system automatically collects and presents information, errors due to manual confirmation are eliminated, and the user can appropriately recognize the work status. In addition, at this time, the system automatically judges tasks to be prioritized and allocates resources such as staff, so that an appropriate work schedule can be created without depending on the individual skill of the user. becomes possible. At this time, for example, if the system selects and presents equipment that should be prioritized for installation in order for the system to perform its original function, such as providing virtual viewpoint content, the original work schedule could not be completed. Also, it is possible to operate the minimum system. Note that the multi-camera operation system may be used for generating virtual viewpoint images, but may be a system used for purposes other than generating virtual viewpoint images. Here, in the present embodiment, "image" includes still images and moving images (video).
In this embodiment, a multi-camera operation system will be described as an example, but the present invention is not limited to this. In other words, for any system that needs to prepare a large number of equipment (various processes, including installation of equipment, to enable the system to perform its prescribed functions), remove equipment, etc. , the following discussion can be applied.

FIG. 1 shows a configuration example of a multi-camera operation system 100 according to this embodiment. The multi-camera operation system 100 includes, in one example, sensor systems 110 a - 110 z, a switching hub 120 , a camera operation terminal 130 , an image computing server 140 , a controller 160 and an image processing system 180 . Although FIG. 1 describes an example in which there are 26 sensor systems 110a-110z, more than 26 sensor systems may be used, or less than 26 sensor systems may be used.

Controller 160 includes control station 161 . The control station 161 manages the operating states of the blocks that make up the multi-camera operation system 100 and controls parameter setting, etc., through the daisy chains 121a-121y, the networks 122a-122b, and 123-124.

First, the configuration and operation of the sensor systems 110a-110z will be described. The sensor systems 110a-110z have microphones 111a-111z, cameras 112a-112z, pan heads 113a-113z, external sensors 114a-114z, and batteries 115a-115z, respectively. The sensor systems 110a-110z also have camera adapters 116a-116z, respectively, and are interconnected by daisy chains 121a-121y via the camera adapters 116a-116z. However, this configuration is an example, and the sensor systems 110a to 110z may have configurations other than these. Also, in FIG. 1, the sensor systems 110a to 110z have a common configuration, but at least some of the sensor systems may have a different configuration from the other sensor systems.

In this embodiment, the sensor systems 110a to 110z are simply referred to as the "sensor system 110" unless otherwise specified. The components of the sensor system 110 are also referred to as a microphone 111, a camera 112, a camera platform 113, an external sensor 114, a battery 115, and a camera adapter 116, unless otherwise distinguished. The sensor system 110 obtains image data by photographing with a camera 112 possessed by each, and obtains audio data by recording using a microphone 111 . That is, the multi-camera operation system 100 can obtain image data captured from multiple directions by using multiple sensor systems 110 that capture the same subject from different directions. Similarly, the multi-camera operation system 100 uses these multiple sensor systems 110 to pick up sounds at different positions (with different directional characteristics as necessary), thereby collecting audio data recorded at multiple positions. can be obtained.

The optical axis of each of the plurality of cameras 112 is fixed, and the imaging range is not moved horizontally or vertically by panning, tilting, or the like. Adjustment of the optical axis at the time of initial setting of the camera 112, correction of deviation of the optical axis due to the environment, and the like can be performed by the platform 113 on which the camera 112 is installed. Each of the plurality of sensor systems 110 transmits image data obtained by photographing to other sensor systems downstream thereof via daisy chains 121a to 121y. At this time, each sensor system 110 also transfers image data acquired from other upstream sensor systems to other downstream sensor systems. For example, the sensor system 110a transmits image data captured by the camera 112a to the camera adapter 116b of the sensor system 110b via the daisy chain 121a after performing image processing described later in the camera adapter 116a. Similarly, sensor system 110b transmits image data captured using camera 112b along with image data obtained from sensor system 110a (possibly in separate signals) to sensor system 110c. Note that the sensor system 110z transmits image data obtained by photographing and image data obtained from an upstream sensor system to the switching hub 120 via the network 122b. To prevent an increase in the number of connection cables and the complexity of wiring work by using a daisy chain even if the image data capacity increases due to the increase in the resolution of captured images to 4K, 8K, etc. and the increase in the frame rate. can be done.

The same applies to handling of a plurality of microphones 111 and audio data picked up by those microphones 111 . In other words, it is assumed that the multiple microphones 111 have fixed directivity directions and do not move in that direction. Also, audio data obtained by sound collection is transmitted via a daisy chain. Through such operations, the image data and audio data acquired by the sensor systems 110 a - 110 z are transmitted from the sensor system 110 z to the switching hub 120 using the network 122 b and then transferred to the image computing server 140 .

The external sensor 114 acquires various types of environmental information by methods other than photographing and collecting sound. For example, the external sensor 114 can acquire various information such as vibration and tilt of the sensor system 110, temperature, humidity, wind direction, and wind speed. Note that these are just examples, and the external sensor 114 can have any device configuration for acquiring any environmental parameter.

The battery 115 is used when executing a workflow for installing or removing the sensor system 110 at a facility such as a stadium, which will be described later. It should be noted that the sensor system 110 may not use the battery 115 after the system power supply is installed after installation is complete. Further, the sensor system 110 may be configured to operate using a power source without the battery 115 if the power source for the system is pre-installed. It should be noted that a permanent power source installed in the facility may be used instead of the system power source.

Although FIG. 1 shows an example in which the camera 112 and the camera adapter 116 are separated, they may be one device contained in the same housing. Also, the microphone 111 may be a microphone built into the camera 112, or a separate device may be prepared as shown in FIG. In this way, multiple functional units shown in FIG. 1 may be implemented by one device, and one functional unit may be implemented by multiple devices working together.

In addition, in this embodiment, the term "image" includes concepts of both moving images and still images unless otherwise specified. That is, the multi-camera operation system 100 can handle both still images and moving images. Also, in this embodiment, a case where a virtual viewpoint image and a virtual listening point sound are provided by the multi-camera operation system 100 as virtual viewpoint content will be described, but the present invention is not limited to this. For example, virtual viewpoint content may not include sound. Also, for example, the sound included in the virtual viewpoint content may be provided in a form other than the virtual listening point sound, such as the sound recorded by the microphone closest to the virtual viewpoint. Also, in the present embodiment, the description of audio is partially omitted for the sake of simplicity, but it is assumed that both images and audio are processed.

Next, the configuration and operation of image computing server 140 will be described. Image computing server 140 processes data obtained from sensor system 110z. Image computing server 140 includes time server 141 and front end server 142 .

The time server 141 has a function of distributing time and synchronization signals, and distributes time and synchronization signals to the sensor systems 110a to 110z via the switching hub 120. FIG. When the sensor systems 110a-110z receive the time and synchronization signals via the camera adapters 116a-116z, they synchronize the cameras 112a-112z based on the time and synchronization signals to establish image frame synchronization. That is, the time server 141 synchronizes the photographing timings of the multiple cameras 112 .

Front-end server 142 reconstructs and converts the data format into segmented transmission packets from the images and audio acquired from sensor system 110z. After that, it is written in a database (not shown) in the front-end server 142 according to the camera identifier, the data type indicating whether it is an image or an audio, and the frame number.

Note that the front-end server 142 temporarily stores the data acquired from the camera adapter 116 in the DRAM, and buffers the data until the foreground image, background image, audio data, and three-dimensional shape data are complete. In the following description, the foreground image, background image, audio data, and three-dimensional shape data are collectively referred to as shooting data. Meta information such as routing information, time code information (time information), and camera identifier is added to the photographed data, and the front-end server 142 checks the attributes of the data based on this meta information. Accordingly, the front-end server 142 can determine whether or not the photographed data is complete by checking the buffered data for a certain time. According to this determination, when data is transferred from the camera adapter 116 via the network 122 or the like, even if the order in which the network packets are received does not correspond to the order in which the data is received, by waiting until the necessary data is ready, File generation can be performed reliably. Note that the background image may be captured at a frame rate different from that of the foreground image. For example, if the frame rate of the background image is 1 fps [frames per second], one background image is acquired every second. On the other hand, when the foreground image is acquired at, for example, 60 fps, there may be timings at which the foreground image is acquired but the background image is not acquired. Therefore, at such timing, the front-end server 142 may determine that all the data are complete without the background image.

As shown in FIG. 11, the camera operation terminal 130 is, for example, a processing device having a CPU 1101, a memory (RAM, ROM) 1102, a graphic memory 1103, and an interface 1104, such as a general-purpose personal computer (PC). sell. Here, the interface 1104 can include a wireless communication interface 1105, an image input/output interface 1106, and an audio input/output interface 1107, for example. These configurations are merely examples, and for example, the camera operation terminal 130 can display a predetermined screen on an external display device such as an HMD (Head Mounted Display) via the image input/output interface 1106. It may have a built-in display. Also, the camera operation terminal 130 has a camera as an image input/output interface 1106, for example, and captures the state of work with the camera, imports it into the camera operation terminal 130, and shares it with other camera operation terminals 130 as necessary. You may Also, as the image input/output interface 1106, a function of communicating with an external scanner for importing information such as drawings may be used. good too. Similarly, the camera operation terminal 130 can cause an external speaker to output predetermined audio via the audio input/output interface 1107, but such a speaker may be incorporated. Also, the camera operation terminal 130 may include a microphone as the audio input/output interface 1107, capture audio data into the camera operation terminal 130, and share it with other camera operation terminals 130 as necessary. The camera operation terminal 130 differs from the configuration shown in FIG. 11 in that a processor other than the CPU 1101 reads a program from a storage device such as a memory and executes processing, or that dedicated hardware executes processing described later. may have a configuration.

The camera operation terminal 130 is configured to be able to communicate with the sensor system 110 via the wireless communication interface 1105, for example. The camera operation terminal 130 can operate an operation system application by executing a program stored in the memory 1102 by the CPU 1101, for example. The operation system application is used when operating the multi-camera system, and is used, for example, to manage the operation from planning of event shooting, installation of the system, shooting, and withdrawal of the system. In addition, the camera operation terminal 130 can be configured to be capable of displaying screens of operation system applications necessary for operating the system, and communicating with other camera operation terminals using text messages, voice, and the like.

The image processing system 180 performs image processing on image data obtained by photographing, and performs processing up to screen display. The functional configuration of this image processing system 180 will be described with reference to FIG. Since the image processing system 180 cooperates with the front-end server 142 described above, part of the functional configuration of the front-end server 142 is also illustrated in FIG. The front-end server 142 has a multi-viewpoint image storage unit 200 and a real space information storage unit 210 as its functions. As described above, the multi-viewpoint image holding unit 200 holds image data related to images synchronously photographed by a plurality of cameras installed to surround a competition field, studio, or the like. Also, the real space information holding unit 210 holds information about a space of a predetermined range including the subject. The real space information holding unit 210 stores three-dimensional shape data of an object (background object) reflected as a background in a virtual viewpoint image, such as a stadium field, spectator seats, or studio equipment, and a virtual viewpoint. Holds three-dimensional space information, etc. that indicates the range that can be set.

The image processing system 180 includes an image processing device 220 and a virtual camera operation terminal 230 . The image processing device 220 has, for example, a virtual viewpoint image generation unit 221, a virtual camera path calculation unit 222, a virtual camera information storage unit 223, a display image generation unit 224, and a device communication unit 225. The virtual camera operation terminal 230 also has a terminal communication unit 231, an image display unit 232, a virtual camera path instruction unit 233, and a user information transmission unit 234, for example.

The virtual viewpoint image generation unit 221 uses the multi-viewpoint images acquired from the multi-viewpoint image storage unit 200 to generate three-dimensional shape data of the foreground object. The generated foreground 3D shape data and the background 3D shape data acquired from the real space information storage unit 210 are mapped with a texture that matches the virtual viewpoint acquired from the virtual camera path calculation unit 222, and rendering is performed. Thus, a virtual viewpoint image is generated. In the process of generating the virtual viewpoint image, the virtual viewpoint image generation unit 221 calculates the coordinates of the foreground object and the background object reflected in the virtual viewpoint image to be generated, and performs texture mapping and rendering only for the coordinates. Run. The virtual viewpoint image generating unit 221 outputs the calculated coordinate values to the virtual camera information holding unit 223, which will be described later, as subject foreground coordinates and real space background coordinates.

The virtual camera path calculation unit 222 calculates temporally continuous virtual camera path parameters based on instructions from the user to the virtual camera path instruction unit 233 described later. A virtual camera path parameter is a parameter indicating the movement of a virtual camera when a user observes an object, and includes at least the position, orientation (gazing direction), and angle of view (focal length) of the virtual camera. Note that the virtual camera path parameter is associated with the frame number or time code attached to the multi-viewpoint image so that it can be specified which moment in the shooting scene the virtual camera path parameter relates to. The virtual camera path calculation unit 222 refers to the real space information acquired from the real space information holding unit 210 when calculating the virtual camera path parameters, and sets the virtual camera path within the range in which the virtual viewpoint can be set.

The virtual camera information holding unit 223 accumulates subject foreground coordinates and real space background coordinates acquired from the virtual viewpoint image generation unit 221 and virtual camera path parameters calculated by the virtual camera path calculation unit 222 .

The display image generation section 224 generates a display image to be displayed on the image display section 232 of the virtual camera operation terminal 230 . The display image generated here is a virtual viewpoint image generated by the virtual viewpoint image generation unit 221 . The device communication unit 225 is a functional unit for performing wired communication or wireless communication via a network (not shown) or the like between the image processing device 220 and the virtual camera operation terminal 230 . Between the image processing device 220 and the virtual camera operation terminal 230, image, voice, and text data and instruction information such as a virtual camera path instruction sent from the user side when generating a virtual viewpoint image are transmitted and received. be.

In the virtual camera operation terminal 230, the terminal communication section 231 transmits and receives various information to and from the device communication section 225 as described above. The image display unit 232 displays the image generated by the display image generation unit 224 described above and received via the terminal communication unit 231 on the screen. The virtual camera path instruction unit 233 receives an instruction for the virtual camera path from the user, and transmits information indicating the instruction to the image processing device 220 using the terminal communication unit 231 . The image processing device 220 receives this information using the device communication section 225 and passes the received information to the virtual camera path calculation section 222 . Note that the user does not have to specify exactly all of the virtual camera parameters for the entire time they wish to view. For example, the user may want to "view a virtual viewpoint image focusing on a specific subject (a player, performer, etc.)", "want to view a certain range around the ball", or "see a more noteworthy event occurring." You may input the instruction|indication based on various viewpoints, such as wanting to watch the part where it is. The user information transmission unit 234 attaches user information such as a user ID to information transmitted from the terminal communication unit 231 to the device communication unit 225 .

The virtual viewpoint content in this embodiment is content that includes a virtual viewpoint image as an image obtained when an object is photographed from a virtual viewpoint. In other words, the virtual viewpoint image is an image corresponding to the appearance at the designated viewpoint. A virtual viewpoint (virtual viewpoint) may be specified by the user, or may be automatically specified based on the result of image analysis or the like. That is, the virtual viewpoint image includes an arbitrary viewpoint image (free viewpoint image) corresponding to a viewpoint arbitrarily designated by the user. The virtual viewpoint image also includes an image corresponding to a viewpoint specified by the user from among a plurality of candidates and an image corresponding to a viewpoint automatically specified by the device.

Next, the functional configuration of the camera adapter 116 will be described using FIG. Camera adapter 116 includes network adapter 300 , transmission section 310 , image processing section 320 , and external device control section 330 . Here, the network adapter 300 includes a data transmission/reception section 301 and a wireless communication section 302 , and the transmission section 310 includes an image/sound transmission processing section 311 and a data compression/decompression section 312 . The image processing unit 320 also includes a foreground/background separation unit 321 , a three-dimensional shape data generation unit 322 , and a calibration control unit 323 . The external device control section 330 also includes a microphone control section 331 , a camera control section 332 , a camera platform control section 333 , and a sensor control section 334 . External device control section 330 further includes external device connection state detection section 335 .

The network adapter 300 includes various functions for communicating with other devices. The data transmitter/receiver 301 performs data communication with other camera adapters 116 , front end servers 142 , time servers 141 and control stations 161 via the daisy chain 121 , network 123 and network 124 . For example, the data transmission/reception unit 301 outputs to another camera adapter 116 a foreground image and a background image obtained by separating an image captured by the camera 112 by a foreground/background separation unit 321 described later. The output destination camera adapter 116 is the adjacent downstream camera adapter 116 in the daisy chain 121 among the camera adapters 116 of the multi-camera operation system 100 . A number of camera adapters 116 output foreground images and background images, so that the foreground images and background images shot from a plurality of viewpoints are transmitted to the front-end server 142, and virtual viewpoint images are generated by the image processing system 180. be done. Note that there may be a camera adapter 116 that outputs the foreground image separated from the captured image and does not output the background image.

A wireless communication unit 302 communicates with the wireless communication interface 1105 of the camera operation terminal 130 . The data communicated via the wireless communication unit 302 is data processed by the external device connection state detection unit 335 , which will be described later, and is data for executing processing by the operation system application running on the camera operation terminal 130 .

The transmission unit 310 has a function of executing signal processing on data transmitted and received via the network adapter 300 . The image/audio transmission processing unit 311 creates a message for transferring image data or audio data to another camera adapter 116 or the front end server 142 via the data transmission/reception unit 301 . The message includes image data or audio data and meta information of each data. Note that the meta information of the present embodiment includes a time code or sequence number at the time of image shooting or sound sampling, a data type, an identifier indicating an individual microphone 111 or camera 112, and the like. Also, the image data or audio data to be transmitted may be data-compressed by the data compression/decompression unit 312 . The image/audio transmission processing unit 311 also receives messages from other camera adapters 116 via the data transmission/reception unit 301 . Then, the image/audio transmission processing unit 311 restores the image data or the audio data from the data divided into the packet size specified by the transmission protocol according to the data type included in the message. Note that when the data is restored, if the data is compressed, the data compression/decompression unit 312 performs decompression processing. A data compression/decompression unit 312 compresses data transmitted/received via the data transmission/reception unit 301 by applying a predetermined compression method, compression ratio, and frame rate, and decompresses the compressed data.

The image processing unit 320 performs image processing on the image data captured by the camera 112 under the control of the camera control unit 332 and the image data received from the other camera adapter 116 . A foreground/background separation unit 321 separates image data captured by the camera 112 into a foreground image and a background image. That is, the foreground/background separation unit 321 in each of the plurality of camera adapters 116 extracts a predetermined area from the captured image captured by the corresponding camera 112 among the plurality of cameras 112 . The predetermined area is, for example, an area where an object is detected by object detection processing on the captured image. The foreground/background separation unit 321 sets an image within a predetermined region extracted from the captured image as a foreground image, and an image of a region outside the predetermined region (that is, an image of a region other than the foreground image) as a background image. and background image. The three-dimensional shape data generation unit 322 uses the foreground image separated by the foreground/background separation unit 321 and the foreground image received from the other camera adapter 116 to create three-dimensional shape data using, for example, the principle of a stereo camera. Generate image information about The calibration control unit 323 performs color correction processing on the input image to suppress variations in color from camera to camera, image alignment processing for camera shake caused by camera vibration, image clipping processing, and the like. I do.

The external device control section 330 has a function of controlling devices connected to the camera adapter 116 . The camera control unit 332 is connected to the camera 112 in the sensor system including the camera adapter 116, and performs control of the camera 112, acquisition of captured images, provision of synchronization signals, time setting, and the like. The control of the camera 112 indicates, for example, the setting and reference of shooting parameters (number of pixels, color depth, frame rate, white balance, etc.), and the state of the camera 112 (shooting, stopped, synchronizing, error, etc.). Acquisition of information, start and stop of shooting, focus adjustment, etc. are included. The microphone control unit 331, pan head control unit 333, and sensor control unit 334 are connected to the microphone 111, pan head 113, and external sensor 114 in the sensor system including the camera adapter 116, respectively, and control the connected devices. Control. In addition, the microphone control unit 331, camera platform control unit 333, and sensor control unit 334 perform processing for reading information indicating the state of connected devices, such as whether they are operating normally, from each device. The external device connection state detection unit 335 has a function of reading the connection state of the device connected to the camera adapter 116 and the self-check results such as whether each device is operating normally. The external device connection state detection unit 335 communicates with the microphone control unit 331, the camera control unit 332, the camera platform control unit 333, and the sensor control unit 334, and receives the connection state of the external device and the self-check result acquired from each function unit. , is stored in the memory in the external device connection state detection unit 335 . Note that the external device connection state detection unit 335 detects only the connection state of the cable when power is not supplied to the external device. Cable connection can be detected at the hardware level by setting one of the cable terminals of the camera adapter 116 to a HIGH port so that it goes LOW when the cable is connected.

(Processing flow)
Next, a workflow that is executed when a plurality of cameras 112 and microphones 111 are installed in a facility such as a stadium, a stadium, a concert hall, etc., and shooting is performed will be described. FIG. 4 schematically shows an overview of the workflow. Note that the processing described below is realized under the control of the camera operation terminal 130 unless otherwise specified. Note that the processing executed by the camera operation terminal 130 is realized, for example, by the CPU 1101 of the camera operation terminal 130 executing a program stored in the memory 1102 .

In this workflow, first, before a series of processes, an operator (user) who installs and operates the multi-camera operation system 100 collects necessary information (advance information) before installation and performs installation such as planning. Perform pretreatment. Through this pre-installation processing, the camera operation terminal 130 receives settings according to the plan from the user who has planned how to install and dismantle the system based on the prior information (S401).

The advance information here includes, for example, facility information, equipment information, and operation information. For example, if the shooting location is a stadium, the facility information is stadium information, and includes information such as the shape of the stadium, sound, lighting, power supply, transmission environment, and three-dimensional shape data of the stadium. The equipment information includes, for example, information about photographing equipment such as a camera, pan head, lens, and microphone, information equipment such as a LAN, PC, server, and cable, and relay vans. It should be noted that not all of this information must necessarily be entered. Note that the camera operation terminal 130 receives an input regarding arrangement information of the camera, camera platform, and microphone among the shooting equipment for which the equipment information has been input. Layout information can be input using the three-dimensional shape data of the stadium described above. The operation information includes information such as camera installation form (work at height, etc.), work type (work stage), work time, number of workers, skill level of workers, and the like. Note that the arrangement information may include information such as a shooting target, shooting time, camera work, and gaze point. For example, depending on the sport type, such as whether the shooting target is a soccer game, the gaze point is changed and the constraint condition of the camerawork is changed. The camera operation terminal 130 manages and modifies a setting information table composed of combinations of these pieces of operation information, and uses it for processing various instructions and the like.

Subsequently, the user installs the equipment in the target facility. Then, each device of the multi-camera operation system 100 executes processing for system operation confirmation according to the command issued from the camera operation terminal 130 based on the user's operation (S402). Since the system does not operate completely while the equipment is being installed, the camera operation terminal 130 and the camera adapter 116 sequentially communicate with each other to perform operation management at the time of installation. This processing will be described later. After equipment installation is complete, calibration of the installed camera 112 is performed. Adjustment of the camera 112 includes angle of view matching and color matching. This adjustment is performed for all installed cameras 112 . Note that the adjustment may be performed based on a user's operation, or may be realized by an automatic adjustment function. Also, in adjusting the angle of view, zoom, pan, tilt, and focus adjustments are executed in parallel, and the camera operation terminal 130 acquires and stores these adjustment results.

Then, before the competition or the like, the user checks the sound collected by the microphone 111 and the image captured by the camera 112, and executes pre-shooting processing that may include adjustment according to the check results (S403). ). After that, the camera operation terminal 130 causes the microphone 111 to collect sound and causes the camera 112 to shoot (S404). It should be noted that although the photographing in S404 includes sound collection using the microphone 111, the present invention is not limited to this, and only image photographing may be performed. The process of S404 is continuously executed until the setting in S402 is changed or until the shooting ends. If the setting in S402 is to be changed or the shooting is finished, the process proceeds to S405.

In S405, if the user operation to change the settings in S402 and continue shooting is performed (YES in S405), the camera operation terminal 130 advances the process to S406, and the user operation to complete shooting is performed. If so (NO in S405), the process proceeds to S407. Note that this determination does not have to be performed based on the user's operation, and the camera operation terminal 130 may autonomously determine whether or not it is necessary to change the settings.

In S406, the camera operation terminal 130 changes the settings made in S402 to new settings. The content of the change is determined, for example, based on the user operation acquired in S405. If it is necessary to stop shooting in changing the settings in S406, the shooting may be stopped once and restarted after the settings are changed. Also, if there is no need to stop shooting, the setting can be changed while shooting is continued.

In S<b>407 , the camera operation terminal 130 issues an instruction to edit the images captured by the multiple cameras 112 and the sounds collected by the multiple microphones 111 . Note that the process of S407 may be executed in parallel with the process of S404. For example, in the case of distributing sports competitions, concerts, etc. in real time (for example, the image of the competition is distributed during the competition), the data obtained by shooting in parallel without waiting for the end of shooting in S404 is edited in S407. On the other hand, in the case of distributing a highlight image of a sports competition after the competition, editing is executed after the photographing is finished in S405.

After the shooting and editing are finished, a process of withdrawing the equipment is performed (S408). The withdrawal process is to return the facility to the state before installation by removing the equipment installed in S402, for example. Although the above processing is executed in the camera operation terminal 130, the processing may be executed by the controller 160 when the installation is completed and the system is ready for normal operation. That is, the controller 160 may perform the processing after the pre-shooting processing in S403.

Next, the flow of processing when equipment is installed will be described with reference to FIG. Note that the processing in FIG. 5 is realized by executing a program stored in the memory 1102 by the CPU 1101 of the camera operation terminal 130, for example.

First, the camera operation terminal 130 reads state data (S501). The state data is data stored by the external device connection state detection unit 335 of the camera adapter 116 and indicates the connection state of the device connected to the camera adapter 116 . Note that the external device connection state detection unit 335 may acquire information detected by the external device connection state detection unit 335 in another camera adapter 116 . In this case, the external device connection state detection unit 335 collects information on the connection state of the external devices in the plurality of camera adapters 116 within the range where the daisy chain installation is completed, and collects the collected information collectively. The camera operation terminal 130 can be notified. In addition, the external device connection state detection unit 335 detects the camera operation terminal 130 via, for example, an external access point or a local area network that is temporarily installed (or is installed in advance in the facility where the system is constructed). can be sent to According to this, the camera operation terminal 130 can acquire information from a plurality of external device connection state detection units 335 via a small number (for example, one) of access points, and access points near a large number of camera adapters 116 to get the information. After that, the camera operation terminal 130 reads the reference data (S502). The reference data is, for example, data defined especially from the device information and operation information among the advance information described in S401 of FIG. contains information about Then, the camera operation terminal 130 compares the state data read out in S501 and the reference data read out in S502 (S503). Then, the camera operation terminal 130 changes the screen display to status display or progress display (S504). The state display displays the above-described state data, and the progress display displays the progress of work with respect to the reference data.

Also, the camera operation terminal 130 determines whether or not there is a work correction plan based on the result of S503 (S505). If there is a correction plan (YES in S505), the camera operation terminal 130 displays the correction plan (S506). Then, the camera operation terminal 130 displays the specific camera ID, installation position, etc. (S507). Note that if there is no correction plan (NO in S505), the camera operation terminal 130 ends the processing in FIG. 5 without executing the processing in S506 and S507.

For example, assume that there is a work capacity to install 12 cameras per hour, and a plan is drawn up assuming that 60 cameras will be installed in 5 hours. At this time, if it becomes necessary to shorten the work time by 1 hour when 2 hours have passed since the work, it is necessary to install 36 cameras in the remaining 2 hours. However, due to work capacity, only 24 units can be installed. For this reason, the camera operation terminal 130 can change the plan to select and install 24 cameras, for example, from 36 cameras that have not yet been installed, and display that effect in S506.

6A and 6B show examples of status display on the camera operation terminal 130. FIG. The camera operation terminal 130 has an application switch for displaying the status, and displays the status as shown in FIGS. 6A and 6B by operating the switch. The camera operation terminal 130 can send a request command to the camera adapter 116 to update the information displayed on the status display screen. When the camera adapter 116 receives the status information request command, the camera adapter 116 transmits data relating to the connection status and installation status of each cable or self-check result data of each device to the camera operation terminal 130 . As a result, the current installation status is displayed on the camera operation terminal 130. FIG.

FIG. 6A is an example of state display. The horizontal axis shows task names related to the connection/installation status of each camera, and the vertical axis shows data for each camera. Regarding the connection status of each camera, for example, the connection status of the cable between the camera 112 and the camera adapter 116 is displayed by the information in the Status column next to the camera ID. Information in the Status column next to the lens ID indicates whether or not a cable for controlling the lens is connected to the camera adapter 116, for example. It should be noted that the information in the Status column next to the lens ID may indicate whether the lens is ready to be monitored for proper setting, for example by an external sensor 114 . In this case, the information in the Status column next to the lens ID may be information indicating whether or not the external sensor 114 for monitoring the lens is connected to the camera adapter 116, for example. Information indicating the connection state of cables for communication with other sensor systems 110 and the connection state with other external devices can be managed in the form of a table as shown in FIG. 6A. It should be noted that the display regarding the installation status includes not only the cable connection, but also the self-check result of the connected equipment, or the installation status of the camera 112 and the camera adapter 116 to the mounting brackets for attaching to the facility. good too. Also, the status of the cable itself used for connection may be indicated (for example, by information in the Status column next to the cable ID). In one example, when a camera is ready to take pictures, all the information in the row corresponding to that camera is "OK". Note that in at least some of the sensor systems 110, for external devices that are not used, a column indicating the state thereof may not be prepared, or a symbol such as "-" may be described in the column indicating the state. . If the system is installed in a stadium, for example, instead of using the table format shown in FIG. may be displayed.

FIG. 6B is an example of progress display (progress display). In the example of FIG. 6(B), the display is performed so that the scheduled time, the result, and the progress rate can be listed for each work. The results of S503 in FIG. 5 are used for these displays. For example, in the task of installing a camera, the scheduled start time, scheduled end time, and actual start time and end time are displayed. Further, information is described in the work content column so that it can be determined that this task is a task of installing a camera. Then, when all the contents of the task have been completed, "100%", "completed", or the like is displayed in the progress rate column. If the task is in the middle of execution or if the process ends without achieving the initial schedule, a value lower than 100% is entered in the progress rate column. In this case, to indicate that the task is not being executed, for example, in addition to the number of the progress rate, a character string such as "end" may be described. For example, for a task to install 60 cameras, if 54 cameras have been installed, "90%" is written in the progress rate column. % (end)”.

FIG. 7 is a diagram for explaining a display example of a correction plan when it is determined in S505 of FIG. 5 that there is a correction plan. The operation system application running on the camera operation terminal 130 reads out the estimated task completion time of the workflow stored in the camera operation terminal 130 and the connection state data stored in the external device connection state detection unit 335 . Then, in the installation progress status determined by both data, the operation system application determines that the number of installation completed units does not meet the predetermined threshold for the estimated time (it is determined that the scheduled task cannot be completed). ), generate a remediation plan.

The screen display is, for example, a schematic diagram simulating a stadium, as shown in FIG. are displayed in a distinguishable manner. For example, in FIG. 7, a total of six cameras 711a to 711f surrounded by dashed lines 710 in Area N are not installed. Note that these cameras may correspond to, for example, "Cam03" to "Cam05" and "Cam08" to "Cam10" in FIG. 6A. In this state, for example, it is determined that there is only time left to install the remaining two cameras. In this case, for example, two cameras to be installed are selected and displayed according to the modification plan.

Here, an example of priority order data for setting the order of priority of cameras to be installed will be described. The cameras that make up the system include, for example, a camera that acquires images for generating three-dimensional shape data, a camera for human texture generation, and a background texture generation camera for each application. In one example, the priority is set according to the application of this camera. For example, the three-dimensional shape data generation camera is given the highest priority, the person texture camera is given the next highest priority, and the background texture camera is given the lowest priority. FIG. 7 shows an example of a correction plan when it is determined that cameras 711c and 711d should be installed because they are the cameras for human texture with the highest priority. In the example of FIG. 7, the cameras 711c and 711d are circled to indicate that these cameras should be installed. On the other hand, as shown in FIG. 7, the cameras 711a, 711b, 711e, and 711f, which have a low priority and are not selected as installation targets, are marked with an X, so that these cameras are determined not to be installed. It is shown that By sharing such a display with, for example, the camera operation terminal 130 possessed by the staff who performs the installation work, the staff can visually recognize the camera to be installed and perform the installation work without error. can.

Note that the camera operation terminal 130 communicates with, for example, the camera 730 via the wireless communication unit 302 of the camera adapter 116, and collects and displays information on installed cameras via this communication. In this system, since the sensor systems 110 are connected by a daisy chain, the camera operation terminal 130 can acquire information on other connected sensor systems from the camera adapter 116 of one of the sensor systems 110 . When there are multiple users, multiple camera operation terminals 130 may be used, and data to be collected and data to be displayed are shared among these multiple camera operation terminals 130 . For example, the camera operation terminal 130 communicating with the camera 730 can collect information of the camera 750 via the camera 740, and similarly, the camera operation terminal 130 communicating with the camera 750 can collect information of the camera 730. be. Also, each of the camera operation terminals 130 can collect information on the cameras 740 and 760 . It should be noted that data entry and decisions can be controlled by setting operating privileges for specific users. Note that information may be shared via an external network (for example, a temporary local area network provided within equipment that constructs the system). According to this, even if the connection between the sensor systems 110 is interrupted, it is possible to collect the equipment installation status of the entire system.

Note that the correction plan that designates the camera to be preferentially installed may be determined by a method other than the method based on the camera type (use) when generating the virtual viewpoint image for each camera as described above. For example, depending on the position of a camera that has been installed, the priority is higher for cameras that are farther from installed cameras and that are not installed (i.e., areas where no cameras are installed have higher priority). The remediation plan may be determined in such a manner as Also, a camera to be preferentially installed may be set for each gaze point (A Group or B Group).

Here, a functional configuration example of the camera operation terminal 130 will be described with reference to FIG. The camera operation terminal 130 implements the functions shown in FIG. 8 by executing a program read from the memory 1102 by the CPU 1101, for example. The camera operation terminal 130 includes, for example, an arithmetic processing unit 800, a display unit 810, a communication unit 820, and a data storage unit 830. The display unit 810 performs processing for displaying information on a screen (not shown) based on the processing result of the arithmetic processing unit 800 . A communication unit 820 communicates with the camera adapter 116 to exchange data such as connection status. The data storage unit 830 stores various data (tables) such as reference data and connection status.

The arithmetic processing unit 800 includes, for example, a workflow data setting unit 801, a state data collection unit 802, and a plan determination unit 803 in order to execute processing of the operation system application. As described above, the workflow data setting unit 801 stores the workflow information up to the installation of the multi-camera system in a predetermined area, the withdrawal from the predetermined area, or the confirmation of the photographable state. Set up the installation and withdrawal flow. The set data is used as reference data in subsequent processing. The reference data is data defined especially by the device information and the operation information among the advance information.

9A and 9B show examples of tables stored in the data storage unit 830 as the reference data described above. In FIG. 9A, the process ID is a process identifier assigned in association with one process in the workflow, and includes completion time, flag ID list, and list number. The completion time indicates the time when the corresponding process is scheduled to be completed. The flag ID list is associated with the device connection configuration table shown in FIG. 9B, and in each process, designates the target for reading the installation state such as the connection state of the cable of each device. The list No. is information that designates a device from which information is to be read in the process.

The state data collection unit 802 communicates with the camera adapter 116 and collects state data indicating device connection states, installation states, and the like in each workflow. Then, state data collection unit 802 rewrites the contents of the table stored in data storage unit 830 as necessary. FIG. 9C shows an example of a table stored in the data storage unit 830 as state data. List No. is a number for grouping and managing device configurations for each camera. A given list number is associated with identifiers of devices and connected devices such as camera IDs, lens IDs, and cable 1_IDs. Note that the list No. in FIG. 9A and the list No. in FIG. 9C are associated with each other, and the device read out for each process is designated by this list No. The value of each component is rewritten based on the results collected by the state data collection unit 802 .

The plan determination unit 803 compares the flag ID list of the reference data and the constituent elements of the status data to generate at least one of progress data and a correction plan. The generated correction plan is stored in the data storage unit 830, and is read out from the data storage unit 830 to generate display information when the correction plan exists and the screen is switched to display the correction plan. used. It should be noted that a plurality of plans may be generated, and a configuration may be adopted in which a plurality of correction plans are displayed so that the user can select one as appropriate when the screen is displayed.

FIG. 9D shows an example of modified plan data stored in the data storage unit 830. FIG. In one example, the modified plan data can be stored in the form of a table, as shown in FIG. 9(D). Correction plan data is generated including, for example, the same components as the table components of the reference data shown in FIG. 9(A). In one example, in the modified plan data, the list No. (that is, the equipment to be processed in each process) is changed from the reference data according to the state data. FIG. 9(D) shows that the list numbers of processes with process IDs 1003 to 1005, that is, the configuration of the cameras to be installed, has been changed from the reference data of FIG. 9(A). Although the rewriting of the table according to the correction plan described above can be appropriately executed by the arithmetic processing unit 800 based on the state data, the rewriting of the table may be performed based on the user's judgment of the progress.

FIG. 10 is a diagram for explaining a plan different from the correction plan explained using FIG. It is the same as generating a correction plan when the number of installations completed does not meet the predetermined threshold for the estimated time, but if the correction plan here is reallocated so as to concentrate the resources of the work staff, This is an example when it is determined that the process will be completed within the scheduled time.

In the example of FIG. 10, groups of work staff 1071-1073 each include six staff members. At this time, working staff to be relocated to the uninstalled area can be determined using the number of cameras, the total number of working staff, the skills of the working staff, etc. as parameters. At this time, since the location of the work staff is displayed on the screen, the user can determine the work staff to be relocated to the non-installed area. On the other hand, the camera operation terminal 130 may, for example, display the recommendation information of the work staff to be rearranged based on the above parameters by blinking the icon indicating the work staff. In the example of FIG. 10, since the work staff 1073 exists at a position close to the non-installed area 1010, the staff to be recommended as the staff to be recommended can be distinguished from the staff who are not to be recommended. is displayed in

Note that the position of the work staff can be identified based on the positioning results obtained by the GPS terminal owned by the work staff (for example, the camera operation terminal 130 having a GPS function), but the present invention is not limited to this. For example, the position of the staff may be estimated based on the communication history between the camera operation terminal 130 used by the staff and the camera adapter 116 . For example, from the position of the camera adapter 116 that communicated last, a predetermined range determined based on the time when the communication was performed and the current time can be specified as the position where the staff member exists. In addition, the direction in which the staff is moving while working may be estimated from the communication history, and the position of the staff may be estimated based on the information on the direction.

In the above-described embodiments, the installation of the equipment has been mainly described, but the present system can also be applied to the withdrawal of the equipment. At this time, in the example of equipment installation, for example, the number of units whose cable connection status is “connected” was managed, but when equipment is removed, the connection status is managed as “not connected” for the number of units, and the progress A situation can be determined.

Also, similar progress management can be performed for workflows other than the described one. For example, a signal indicating the task completion state is input to the external device connection state detection unit 335 for transportation of equipment, delivery for transporting equipment to the installation position, adjustment of camera posture, laying of network cables, and laying of power cables. By doing so, similar progress management can be executed. A task management switch is provided in the camera adapter 116, and an operator or worker presses the switch when a predetermined task is completed to generate a signal indicating the completion state of the task. can be configured. As a result, a signal can be input to the external device connection state detection unit 335 each time a predetermined task is completed, and appropriate progress management can be performed.

Further, in the above-described embodiment, a correction plan is generated and displayed based on the reference data and the state data, but if the correction plan is not generated, a warning may be given to the user. For example, an alert may be displayed on the screen of the camera operation terminal 130 when a correction plan is not generated (cannot be generated).

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

100: multi-camera operation system, 110: sensor system, 116: camera adapter, 120: switching hub, 130: camera operation terminal, 140: image computing server, 160: controller, 180: image processing system, 800: arithmetic processing unit , 801: workflow data setting unit, 802: status data collection unit, 803: plan determination unit, 810: display unit, 820: communication unit, 830: data storage unit

Claims (21)

a setting means for setting a task relating to the plurality of equipment to bring the system operating the plurality of equipment into a state capable of performing a predetermined function or a task relating to the plurality of equipment for withdrawing the system;
an acquisition means for acquiring information including a connection state of each of the plurality of devices to the system from a device to which the plurality of devices are connected;
control means for causing a display means to display a screen regarding the progress of the task based on the task related to the plurality of devices set by the setting means and the information acquired by the acquisition means;
A processing apparatus comprising: the connection state is a state of whether or not each of the plurality of devices is connected to the device;
The control means causes the display means to display a screen indicating the connection state as a screen regarding the progress.
2. The processing apparatus according to claim 1, characterized by: The information includes results of self-checks performed by each of the plurality of equipment,
The control means causes the display means to display a screen including the result as a screen regarding the progress;
3. The processing apparatus according to claim 1 or 2, characterized in that: The result of the self-check includes a result of checking by each of the plurality of equipment whether each of the plurality of equipment is operating normally.
4. The processing apparatus according to claim 3, characterized in that: The control means causes the display means to display a screen showing a progress rate determined based on the scheduled time to end the task and the information as a screen related to the progress.
5. The processing apparatus according to any one of claims 1 to 4, characterized in that: The setting means sets the task based on a user operation.
The processing apparatus according to any one of claims 1 to 5, characterized in that: wherein said system is a multi-camera system, said plurality of equipment comprising a plurality of cameras;
The processing apparatus according to any one of claims 1 to 6, characterized in that: The control means displays, as a screen relating to the progress of the task, the positions where the plurality of equipment should be installed with respect to the position of the object to be photographed, and the equipment that has completed the work related to the task and the equipment that has not completed the work. causing the display means to display a screen that is distinguishable from the device;
8. The processing apparatus according to claim 7, characterized by: further comprising generating means for generating a modified plan for the task;
When the generating means generates the modified plan, the control means causes the display means to display a screen showing the modified plan.
The processing apparatus according to any one of claims 1 to 8, characterized in that: The generation means generates the correction plan when it is determined that the task cannot be completed by the scheduled time based on the progress.
10. The processing apparatus according to claim 9, characterized by: The generating means selects, from among the plurality of pieces of equipment, a piece of equipment that should preferentially perform work related to the task,
The control means causes the display means to display a screen showing the selected equipment as the screen showing the modification plan.
11. The processing apparatus according to claim 9 or 10, characterized in that: The generating means selects equipment that should be prioritized for work related to the task according to the use of the plurality of equipment.
12. The processing apparatus according to claim 11, characterized by: The generating means generates, as the correction plan, information regarding the relocation of staff working on the task,
The control means causes the display means to display, as a screen showing the correction plan, a screen that displays the staff to be relocated so as to be distinguishable from the staff that are not to be relocated.
13. The processing apparatus according to any one of claims 9 to 12, characterized in that: the system is a multi-camera system, the plurality of equipment includes a plurality of cameras;
The control means causes the display means to display a screen showing the correction plan on the screen showing the positions where the plurality of equipment should be installed with respect to the position of the imaging target.
14. The processing apparatus according to any one of claims 9 to 13, characterized in that: wherein the system is a multi-camera system for generating virtual viewpoint images;
15. The processing apparatus according to any one of claims 1 to 14, characterized in that: The system is a multi-camera system in which the subject is photographed from multiple directions in order to generate three-dimensional shape data of the subject used for generating the virtual viewpoint image.
16. The processing apparatus according to any one of claims 1 to 15, characterized in that: A control method for a processing device, comprising:
A setting step in which the setting means sets a task relating to the plurality of equipment to bring the system operating the plurality of equipment into a state capable of performing a predetermined function or a task relating to the plurality of equipment for withdrawal of the system. When,
an obtaining step in which an obtaining means obtains information including a connection state of each of the plurality of pieces of equipment to the system from a piece of equipment to which the plurality of pieces of equipment is connected;
a control step in which the control means causes a display means to display a screen regarding the progress of the task based on the task related to the plurality of devices set in the setting step and the information obtained in the obtaining step;
A control method characterized by having the connection state is a state of whether or not each of the plurality of devices is connected to the device;
In the control step, causing the display means to display a screen indicating the connection state as a screen regarding the progress;
18. The control method according to claim 17, characterized by: The information includes results of self-checks performed by each of the plurality of equipment,
In the control step, causing the display means to display a screen containing the result as a screen regarding the progress;
19. The control method according to claim 17 or 18, characterized by: In the control step, causing the display means to display a screen showing a progress rate determined based on the scheduled time to end the task and the information as a screen regarding the progress;
20. The control method according to any one of claims 17 to 19, characterized in that: A program for causing a computer to function as each means of the processing apparatus according to any one of claims 1 to 16.

Images