JP7558652B2 - Placement determination device, placement determination method, and program - Google Patents

Description

The present invention relates to a technique for determining the placement of multiple imaging devices.

In recent years, a technology that uses multiple image capture devices placed in different locations to capture images synchronously and generate a virtual viewpoint image from a specified virtual viewpoint using the multiple images obtained by the capture has been attracting attention.

Patent document 1 describes a technology for generating a virtual viewpoint image, and describes arranging multiple image capturing devices along the circumference so that they face the center of the circle, which is the area to be captured.

JP 2010-79505 A

In Patent Document 1, a difference occurs between the number of imaging devices that capture a subject that exists near the center of a circle, which is the area to be captured, and the number of imaging devices that capture a subject that exists at a position distant from that. Furthermore, when the area to be captured is large, the difference becomes larger. Therefore, there may be a large difference between virtual viewpoints in the reproduction accuracy of the subject in the virtual viewpoint image.

In view of the above problems, the present invention aims to reduce the difference in reproduction accuracy of the subject between virtual viewpoints, even when the shooting area is wide.

One aspect of the present invention is as follows: An arrangement determination device that determines an arrangement of a plurality of image capturing devices used to generate a virtual viewpoint image includes: an area acquisition means that acquires information representing a shooting target area to be captured by the plurality of image capturing devices; a group acquisition means that acquires information on a plurality of groups corresponding to a plurality of directions in which the shooting target area is captured; and an arrangement determination means that determines an arrangement of the plurality of image capturing devices based on the information acquired by the area acquisition means and the information acquired by the group acquisition means such that the entire area of the shooting target area is captured from the first direction by two or more image capturing devices included in the plurality of image capturing devices and belonging to a first group of the plurality of groups corresponding to a first direction , and the entire area of the shooting target area is captured by other two or more image capturing devices included in the plurality of image capturing devices and belonging to a second group of the plurality of groups corresponding to a second direction different from the first direction.

The present invention makes it possible to reduce the difference in accuracy of reproducing a subject between virtual viewpoints, even when the subject area is wide.

FIG. 1 shows an example of the configuration of an image processing system. FIG. 1 is a diagram illustrating an example of a hardware configuration of a placement determination device. FIG. 1 is a diagram showing an example of the configuration of a placement determination device according to a first embodiment; 1 is a flowchart showing an example of a processing procedure of a placement determination device according to a first embodiment. FIG. 1 is a diagram showing an example of a photographing target area according to the first embodiment; Diagram to explain resolution Diagram showing an example of placement area FIG. 1 shows an example of the arrangement of multiple cameras in an arrangement area. Diagram showing ideal multiple camera arrangement Diagram showing an example of multiple camera placement A diagram showing another example of the arrangement of multiple cameras. A diagram showing another example of the arrangement of multiple cameras. A diagram showing an example of the determined layout 11 is a flowchart showing an example of a processing procedure of a placement determination device according to a second embodiment. A diagram explaining the angle of the imaging target area as seen from the placement area. FIG. 13 is a diagram showing an example of the configuration of a placement determination device according to a second embodiment; FIG. 1 is a diagram showing an example of a table to which the arrangement determination device according to the first embodiment refers; FIG. 13 is a diagram showing an example of the configuration of a placement determination device according to a third embodiment; FIG. 13 is a diagram showing an example of a table to which the arrangement determination device according to the third embodiment refers; FIG. 13 is a diagram showing an example of a photographing target area according to the third embodiment; 11 is a flowchart showing an example of a processing procedure of a placement determination device according to a third embodiment.

Below, a description will be given of a mode for carrying out the present invention with reference to the drawings. Note that the following embodiment does not limit the present invention, and not all of the combinations of features described in the present embodiment are necessarily essential to the solution of the present invention. Note that the same components will be described with the same reference numerals.

<Embodiment 1>
1 is a diagram showing an example of the configuration of an image processing system 100. The image processing system 100 includes a multi-camera system 101, a hub 104, a control device 105, and a virtual viewpoint image generating device 106. The image processing system 100 may also include a virtual viewpoint designating device 107 and a placement determining device 108.

The image processing system 100 is a system that generates a virtual viewpoint image that represents the view from a specified viewpoint based on multiple images based on shooting by multiple shooting devices and a specified virtual viewpoint. The virtual viewpoint image in this embodiment is also called a free viewpoint video, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user, and also includes, for example, an image corresponding to a viewpoint selected by the user from multiple candidates. In addition, in this embodiment, the virtual viewpoint is mainly specified by a user operation, but the virtual viewpoint may be automatically specified based on the results of image analysis, etc. In addition, in this embodiment, the virtual viewpoint image is mainly specified as a video, but the virtual viewpoint image may be a still image.

The multi-camera system 101 has multiple cameras (imaging devices) 102aa-102lc that capture images of the target space from multiple directions. The multiple cameras 102aa-102lc belong to one of multiple camera groups 103a-103l. In the following, unless there is a need to distinguish between them, they will be referred to as camera 102 and camera group 103.

Each camera group 103 has one or more cameras 102. Specifically, three cameras, 102aa, 102ab, and 102ac, belong to camera group 103a. However, the number of cameras belonging to camera group 103 is not limited to three, and may be two or less, or four or more. Furthermore, the number of cameras 102 belonging to each camera group 103 may be the same or different.

The images captured by the multiple cameras 102 are sent to the virtual viewpoint image generating device 106 via the hub 104. The control device 105 also controls the capture of the multi-camera system 101 and manages the placement of the multiple cameras 102 in the multi-camera system 101.

The virtual viewpoint image generating device 106 generates a virtual viewpoint image seen from a viewpoint specified by the virtual viewpoint specifying device 107 based on multiple images captured by the multi-camera system 101. The viewpoint information specified by the virtual viewpoint specifying device 107 is information indicating the position and orientation of the virtual viewpoint. Specifically, the viewpoint information is a parameter set including a parameter indicating the three-dimensional position of the virtual viewpoint and a parameter indicating the orientation of the virtual viewpoint in the pan, tilt, and roll directions. Note that the content of the viewpoint information is not limited to the above. For example, the parameter set as the viewpoint information may include a parameter indicating the size of the field of view (angle of view) of the virtual viewpoint. In addition, the viewpoint information may have multiple parameter sets. For example, the viewpoint information may have multiple parameter sets corresponding to multiple frames that constitute a video of the virtual viewpoint image, and may be information indicating the position and orientation of the virtual viewpoint at each of multiple consecutive time points. Note that the virtual viewpoint specifying device 107 is connected to the virtual viewpoint image generating device 106 or the image processing system 100 by wire or wirelessly.

The virtual viewpoint image generating device 106 generates a virtual viewpoint image, for example, by the following method. First, multiple images are obtained by shooting from different directions using multiple shooting devices. Next, from the multiple images, a foreground image in which a foreground area corresponding to a specific object (subject) such as a person or a ball is extracted, and a background image in which a background area other than the foreground area is extracted are obtained. In addition, a foreground model representing the three-dimensional shape of the specific subject and texture data for coloring the foreground model are generated based on the foreground image, and texture data for coloring a background model representing the three-dimensional shape of the background such as a stadium is generated based on the background image. Then, the texture data is mapped to the foreground model and the background model, and rendering is performed according to the virtual viewpoint indicated by the viewpoint information, thereby generating a virtual viewpoint image. The background is a field and a goal post. However, the method of generating a virtual viewpoint image is not limited to this, and various methods can be used, such as a method of generating a virtual viewpoint image by projective transformation of a shot image without using a three-dimensional model.

The placement determination device 108 determines the placement of the multiple cameras 102 in the multi-camera system 101, as described below. This placement determination device 108 may or may not be included in the image processing system 100. The placement determination device 108 is connected to the control device 105 or the image processing system 100 via a wired or wireless connection.

The above-mentioned foreground image and background image may be extracted from the captured image by the virtual viewpoint image generating device 106, or may be extracted by another device, for example, the camera 102. When the foreground image and background image are extracted by the camera 102, each of the multiple cameras 102 may extract both the foreground image and the background image, or some of the multiple cameras 102 may extract the foreground image and the other cameras may extract the background image. The multiple cameras 102 may also include a camera 102 that extracts neither the foreground image nor the background image.

Figure 2 is a diagram showing an example of the hardware configuration of the placement determination device 108. The CPU 201 realizes each function of the information processing device by controlling the entire information processing device using computer programs and data stored in the ROM 202 and RAM 203. Note that the information processing device may have one or more dedicated hardware components different from the CPU 201, and at least a part of the processing by the CPU 201 may be executed by the dedicated hardware components. Examples of the dedicated hardware components include an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor). The ROM 202 stores programs that do not require modification.

The RAM 203 temporarily stores programs and data supplied from the auxiliary storage device 204, and data supplied from the outside via the communication I/F 207. The auxiliary storage device 204 is formed, for example, of a hard disk drive, and stores various data such as image data and audio data. The display unit 205 is formed, for example, of an LCD display or LED, and displays a GUI (Graphical User Interface) for the user to operate the information processing device.

The operation unit 206 is composed of, for example, a keyboard, mouse, joystick, touch panel, etc., and inputs various instructions to the CPU 201 in response to operations by the user. The communication I/F 207 is used for communication with devices external to the information processing device. For example, when the information processing device is connected to an external device via a wired connection, a communication cable is connected to the communication I/F 207. When the information processing device has a function for wireless communication with an external device, the communication I/F 207 is equipped with an antenna. The bus 208 connects each part of the information processing device to transmit information.

In this embodiment, the display unit 205 and the operation unit 206 are assumed to exist inside the placement determination device 108, but at least one of the display unit 205 and the operation unit 206 may exist as a separate device outside the placement determination device 108. In this case, the CPU 201 may operate as a display control unit that controls the display unit 205 and an operation control unit that controls the operation unit 206.

Figure 3 is a diagram showing an example of the functional configuration of the placement determination device 108. The placement determination device 108 has a tolerance acquisition unit 303, a shooting target area acquisition unit 304, a camera group number acquisition unit 305, and a placement determination unit 306. Furthermore, the placement determination device 108 may have an event information storage unit 301 and a venue information storage unit 302. This embodiment will be described using an example in which the placement determination device 108 has the event information storage unit 301 and the venue information storage unit 302. However, the event information storage unit 301 and the venue information storage unit 302 may be included in a device separate from the placement determination device 108, and the placement determination device 108 may acquire the necessary information from that device.

The event information storage unit 301 stores information about the event to be photographed. The event information is, for example, the following information. Information that can identify the type of event or other events, information about the size of the field, information about the range of movement of the subject, information about the density of the subject, information about the purpose of the virtual viewpoint image, information about the scale of the event, etc. The types of events are not particularly limited, and include sports competitions such as rugby, soccer, bouldering, baseball, track and field, water sports, and ice sports, as well as concerts and plays. The field refers to a surface on which moving objects such as people and balls that are subjects in an event and tools used in the event can move. The range of movement of the subject is set for each event. The density of the subject is a numerical representation of the density of people who are subjects in an event when they are crowded together. For example, when comparing the densities of rugby, soccer, and baseball, rugby has the highest density and baseball has the lowest density. The reason why the density of subjects in rugby is high is because plays such as scrums and mauls occur. Virtual viewpoint images can be used for live broadcasts, archive distribution, behavioral analysis, tactical analysis, and more.

The venue information storage unit 302 stores information about the venue where multiple cameras 102 are placed. The venue information is, for example, the following information: information about the name of the venue, information about the size of the venue, information about structures within the venue where cameras 102 can be placed and their positions, and information about the number of cameras 102 that can be placed. Note that the venue may be a stadium where a rugby or soccer game is held, or a venue for a play or concert. The venue may also be a training ground for a sports team. Information indicating structures where cameras 102 can be placed and their positions may be represented as a two-dimensional image or three-dimensional data.

When the camera 102 captures a predetermined reference subject, the tolerance acquisition unit 303 sets and acquires a lower limit of the resolution, which is the size of the subject in the image. Specifically, the lower limit of the resolution is set depending on the use of the virtual viewpoint image. For example, if the virtual viewpoint image is used as an overhead image for live broadcasting, the lower limit of the resolution is set small, and if the virtual viewpoint image is used as a replay image, the lower limit of the resolution is set large. The lower limit of the resolution will be described later.

The tolerance acquisition unit 303 may store in advance a table of correspondence between the use of the virtual viewpoint image, which is one of the event information, and the minimum resolution, and may set the minimum resolution by referring to the table. In other words, when the tolerance acquisition unit 303 receives an input of information that can identify an event from the operator via the operation unit 206, the tolerance acquisition unit 303 may acquire the minimum resolution by referring to the event information stored in the event information storage unit 301 and the table. An example of the referenced table is shown in FIG. 17(a). Note that the referenced table may be such that the information on the use of the virtual viewpoint image and the information on the minimum resolution are associated with each other, but is not limited thereto. Event information other than the above may be associated with the minimum resolution. Also, the referenced table may be such that the information that can identify an event, such as the name of the event, and the minimum resolution are associated with each other. Also, the referenced table may be stored in the event information storage unit 301, or may be stored in an external device.

The tolerance acquisition unit 303 may also acquire information such as the numerical value of the lower limit resolution directly from the operator via the operation unit 206.

The shooting target area acquisition unit 304 determines the shooting target area, which is the area to be shot by the multi-camera system 101, based on the event information. Specifically, the shooting target area acquisition unit 304 may be configured as follows. That is, first, the shooting target area acquisition unit 304 stores a table in which at least a part of the event information is associated with information representing the shooting target area. The at least a part of the event information is information on the size of the field, information on the range of movement of the subject, etc. Then, when the shooting target area acquisition unit 304 receives input of information that can identify an event from the operator via the operation unit 206, it refers to the event information stored in the event information storage unit 301 and the table, and determines the shooting target area. An example of the referenced table is shown in FIG. 17(b). Note that the referenced table may be, but is not limited to, a table in which information on the size of the field, information on the range of movement of the subject, and information on the shooting target area are associated with each other. Event information other than the above may be associated with information on the shooting target area. In addition, the referenced table may be a table in which information that can identify an event, such as the name of the event, is associated with the shooting target area. Furthermore, the referenced table may be stored in the event information storage unit 301 or in an external device.

The imaging target area acquisition unit 304 may also acquire information about the imaging target area directly from the operator via the operation unit 206.

Here, the shooting target area will be explained. FIG. 5(a) is a diagram showing an example of the shooting target area, and in this diagram, an example is shown when the event is rugby. The shooting target area 503 is a two-dimensional surface, and constitutes a part of the space 504 in FIG. 5(b). This shooting target area 503 is determined by the shooting target area acquisition unit 304 based on the size of the field 501 and the movement range of the subject, such as a person (player) or a ball. Here, the movement range is not only the movement range on the field 501, but also includes the periphery of the field 501, for example, when a throw-in is made by a lineout, because a player stands outside the touchline. The size of the movement range may be set according to the event. For example, in the case of rugby, the movement distance may be defined by the distance or angle from the touchline, dead ball line, touch-in-goal line, goal post 502, etc., or may be defined by a range set to include the field 501. In addition, in the case of soccer, it may be defined by the distance or angle from the touchline, goal line, corner flag position, etc. In baseball, the movement range may be set to include the foul zone.

In this way, the shooting target area 503 is set to include the play area (field 501) and also to include a predetermined range outside the play area. By setting the shooting target area 503 in this way, a high-quality virtual viewpoint image can be generated over the entire field 501.

Figure 5(b) is a diagram showing an example in which a space 504 taking into account the height direction is set as the shooting target area. That is, in the example of Figure 5(b), the shooting target area is a three-dimensional space. In Figure 5(b), the shooting target area is set to include an area including the space above the goal post 502, mainly taking into account the range in which the ball moves. In addition, the shooting target area may be a space taking into account a predetermined height from the field 501. For example, the predetermined height may be the height of a person (player), the height at which a person jumps, or the point reached by a player who is lifted during a lineout.

The camera group number acquisition unit 305 acquires the number of camera groups 103 constituting the multi-camera system 101 so as to capture the target area 503 from multiple directions. Specifically, the camera group number acquisition unit 305 sets the number of camera groups 103 based on the density of the subjects. When the density of the subjects is high, the camera group number acquisition unit 305 sets the number of camera groups 103 to be greater than when the density of the subjects is low. Note that the camera group number acquisition unit 305 may store the correspondence between the density of the subjects, which is one of the event information, and the number of camera groups 103 as a table in advance and set the number by referring to the table. In other words, when the camera group number acquisition unit 305 accepts input of information that can identify an event via the operation unit 206, it may acquire the number of camera groups 103 by referring to the event information stored in the event information storage unit 301 and the table. An example of the table to be referenced is shown in FIG. 17(c). The referenced table may be a table in which information on the density of subjects is associated with information on the number of camera groups 103, but is not limited to this. Event information other than the above may be associated with information on the number of camera groups 103. The referenced table may be a table in which information that can identify an event, such as the name of the event, is associated with the number of camera groups 103. The referenced table may be stored in the event information storage unit 301, or may be stored in an external device.

The camera group number acquisition unit 305 may also acquire information on the number of camera groups 103 directly from the operator via the operation unit 206.

The placement determination unit 306 determines the placement of the cameras 102 based on the lower limit resolution, the shooting target area 503, the number of camera groups, and the venue information. The lower limit resolution is acquired by the tolerance acquisition unit 303, the shooting target area 503 is acquired by the shooting target area acquisition unit 304, and the number of camera groups is acquired by the camera group number acquisition unit 305. The venue information is stored in the venue information storage unit 302. The placement determination unit 306 acquires the necessary venue information from the venue information storage unit 302 based on information for identifying the venue input via the operation unit 206. For example, the acquired venue information includes information on structures within the venue where the cameras 102 can be placed and their positions, information on the number of cameras 102 that can be placed, etc.

Specifically, the placement determination unit 306 first determines the placement positions of the camera groups 103 based on the shooting target area 503, the number of camera groups, and the venue information. Next, the placement determination unit 306 determines the placement of the multiple cameras 102 belonging to each camera group 103. At that time, the placement determination unit 306 determines the placement of the multiple cameras 102 belonging to each camera group 103 so that the entire shooting target area 503 is photographed by the multiple cameras 102 belonging to the camera group 103. Furthermore, the placement determination unit 306 determines the placement of the multiple cameras 102 so that the resolution of the multiple cameras 102 belonging to the camera group 103 exceeds the lower limit resolution in the entire shooting target area 503. Note that, although the placement of the multiple cameras 102 is determined so that the above conditions regarding the shooting target area and the resolution conditions are satisfied in all camera groups 103, this is not limited to this. For example, the placement of the multiple cameras 102 may be determined so that only the multiple cameras 102 belonging to a predetermined number of camera groups 103 satisfy the above conditions. The arrangement of the multiple cameras 102 belonging to a camera group 103 may also be determined so that the camera group 103 in a predetermined position or the camera group 103 in a predetermined relationship satisfies the above conditions. In other words, the multiple cameras 102 belonging to some camera groups 103 may be arranged so that the multiple cameras 102 capture a part of the shooting target area 503, rather than the entire area. Furthermore, the resolution of the multiple cameras 102 belonging to some camera groups 103 does not have to exceed the lower limit resolution in some areas of the shooting target area 503.

Figure 4 is a flowchart showing an example of the processing procedure of the placement determination device 108. The processing performed by the placement determination device 108 will be explained using Figure 4.

First, the placement determination device 108 receives a placement determination instruction from the operator via the operation unit 206. In accordance with the instruction, in step S401, the tolerance acquisition unit 303 acquires the lower limit resolution.

Now, we will explain the minimum resolution, but before that, we will explain the resolution. The resolution here refers to the size of the reference subject in the image captured by the camera 102, and is expressed in terms of the number of pixels. The reference subject may be a typical subject or a virtual subject in the event being photographed, or it may be a specific subject or any subject. For example, a player standing upright and 2m tall is defined as the reference subject. In that case, the number of pixels from the head to the toes of a player standing upright and 2m tall in the image captured by the camera 102 is the resolution.

Figure 6 is a diagram for explaining resolution. The number of pixels in a captured image representing the height of a reference subject (here, a player standing upright and 2 m tall) at position 602a on the field 501 is defined as resolution 601a. The number of pixels in a captured image representing the height of the reference subject at position 602b is defined as resolution 601b. The resolution differs depending on the distance from the camera 102 and the focal length of the camera 102. For example, the resolution 601a at position 602a closer to the camera 102 is greater than the resolution 601b at position 602b further away.

The lower limit resolution is the allowable value of the resolution in the image captured by the camera 102. The lower limit resolution affects the image quality of the subject in the virtual viewpoint image. In other words, if the number of pixels of the reference subject in the virtual viewpoint image becomes larger than the lower limit resolution, image quality degradation occurs due to an effect similar to that of digital zoom. Therefore, the lower limit resolution is determined according to how large the subject is desired to be captured in the virtual viewpoint image and the composition desired. For example, consider capturing the reference subject enlarged to a size that is one-quarter of the size of the virtual viewpoint image. The size of the virtual viewpoint image is 1920 pixels x 1080 pixels. Then, the lower limit resolution is determined to be 1080 pixels ÷ 4 = 270 pixels. The lower limit resolution may be determined to allow a certain degree of digital zoom. For example, if it is determined that up to 1.2 times digital zoom is allowable, 270 ÷ 1.2 = 225 pixels is determined to be the lower limit resolution.

This minimum resolution indicates how many pixels are required to represent a reference subject at any position in the field 501 in the captured image. However, this is not limited to this. For example, the minimum resolution may indicate how many pixels are required to represent a reference subject at a specified position or within a specified range in the captured image, rather than at any position in the field 501.

As described above, the tolerance acquisition unit 303 determines and acquires the lower limit resolution based on information capable of identifying an event input by the operator via the operation unit 206 and a pre-stored table.

Next, in step S402, the imaging target area acquisition unit 304 determines and acquires the imaging target area 503. Specifically, as described above, the imaging target area 503 is determined based on the information capable of identifying the event input by the operator via the operation unit 206 and a pre-stored table. Note that the imaging target area 503 is as described above.

In step S403, the camera group number acquisition unit 305 determines and acquires the number of camera groups 103. The number of camera groups 103 is determined according to the number of directions required for the virtual viewpoint image generation device 106 to generate three-dimensional shape data of the subject with high accuracy. The number of directions required to generate three-dimensional shape data of the subject varies depending on the density of the subject. The camera group number acquisition unit 305 associates the density of the subject with the number of required directions (i.e., the number of camera groups) and stores them in advance as a table. For example, the camera group number acquisition unit 305 generates a virtual viewpoint image using CG (computer graphics) that imitates the subject, and stores the number of required directions and their viewpoint positions in advance as a table, in association with the density of the subject, based on the image quality. For example, in a situation where a subject exists alone, such as a baseball pitcher, occlusion is unlikely to occur, so shooting from a few directions is sufficient, but in a situation where a large number of subjects are crowded together, such as a rugby scrum, shooting from many directions is necessary. The number of directions determined here is determined as the number of camera groups 103. In the following, as an example, it is assumed that the number of camera groups 103 is determined to be 12. The placement area of the camera groups 103 may also be determined in this step, but in this embodiment, it is determined in the next step S404.

In step S404, the placement determination unit 306 determines placement areas for the camera groups 103 based on the number of camera groups and the venue information. The placement determination process will be described in detail with reference to Figs. 7 to 12. Fig. 7 is a diagram showing an example of placement positions for the camera groups 103. First, areas in which the cameras 102 can be placed are determined as candidates for placement areas 701a to 701l based on venue information, such as a blueprint of the stadium 702. Candidates for placement areas 701a to 701l include the pillars, aisles, handrails, and ceiling of the stadium 702. The determined candidate placement areas may be displayed on the display unit 205. In the following, unless there are special circumstances, placement areas 701a to 701l will be referred to as placement area 701.

Next, from among the candidates for the placement area 701, the placement area 701 of the camera group 103 is determined according to the criteria described below. If the interval between two adjacent camera groups 103 that capture the subject becomes wider, the image quality of the virtual viewpoint image generated from the virtual viewpoint specified between them deteriorates. Therefore, if the interval between the camera groups 103 is uneven, the image quality of the virtual viewpoint image varies depending on the position of the specified virtual viewpoint. Therefore, it is preferable to determine the placement areas 701 of the camera groups 103 so that they are as evenly spaced as possible when viewed from the subject of the shooting target area 503. For example, the placement areas 701 are determined so that the angles between the adjacent placement areas 701 are equal when viewed from the subject at the center 703 of the shooting target area 503. When the number of camera groups 103 is 12, the ideal angle between the adjacent placement areas 701 is 360 degrees/12=30 degrees. Therefore, for example, the placement areas 701a and 701b are determined so that the angle 704ab between them is 30 degrees. However, due to structural reasons of the stadium 702, such as the position of pillars, it may be impossible to determine the placement areas 701 at equal intervals. In this case, the placement areas 701 may be determined based on a predetermined standard, such as setting an allowable range of 1.5 times the ideal angle. In other words, the placement areas 701 are determined so that the angle between adjacent placement areas 701 as seen from the subject does not exceed a predetermined value.

In step S405, the placement determination unit 306 determines the number and placement of the two or more cameras 102 belonging to the camera group 103 for each camera group 103 based on the placement area 701, the lower limit resolution, and the shooting target area 503 of the camera group 103. At that time, the number and placement of the two or more cameras 102 belonging to the camera group 103 are determined so as to satisfy the following conditions. That is, the number and placement of the two or more cameras 102 are determined for each camera group 103 so that the entire shooting target area 503 is shot. Furthermore, the number and placement of the cameras 102 are determined so that the resolution exceeds the lower limit resolution in the entire shooting target area 503. In addition, the placement is determined so that the number of cameras 102 is as small as possible. Note that the placement of the cameras 102 here includes the position and orientation of the cameras 102.

First, the positions at which the multiple cameras 102 are placed within the placement area 701 will be described with reference to FIG. 8. The multiple cameras 102 belonging to the camera group 103 are placed in the corresponding placement area 701. For example, the multiple cameras 102 belonging to the camera group 103a are placed in the corresponding placement area 701a. Within the placement area 701, the multiple cameras 102 belonging to the camera group 103 are placed as close as possible.

Figure 9 shows an example of an ideal situation in which three cameras 102aa, 102ab, and 102ac in the same camera group 103a are arranged so that their optical centers overlap. The shooting range of camera 102aa is 901aa. The shooting range of camera 102ab is 901ab. The shooting range of camera 102ac is 901ac. The shooting range 901 of camera 102 spreads radially. Therefore, when the optical centers overlap, cameras 102aa, 102ab, and 102ac can be arranged so that the shooting ranges 901aa, 901ab, and 901ac are in contact. By having the shooting ranges 901 touching without overlapping, a wide area can be covered efficiently with the three cameras 102aa to 102ac. However, in reality, due to physical constraints of camera 102, it is not possible to arrange them so that their optical centers overlap. Therefore, by placing them as close as possible, overlap of the shooting ranges 901 is reduced.

Returning to FIG. 8, for example, multiple cameras 102ha, 102hb, and 102hc belonging to camera group 103h are positioned within a range of a predetermined angle 801h when viewed from the center 703 of the shooting target area 503. For example, multiple cameras 102 belonging to camera group 103 are positioned within a predetermined distance.

From another perspective, each camera 102 is placed closer to the cameras 102 in the camera group 103 to which it belongs than to the cameras 102 in the other camera groups 103. For example, let us consider adjacent camera groups 103g and 103h. Assume that cameras 102ga, 102gb, and 102gc belong to camera group 103g. Also assume that cameras 102ha, 102hb, and 102hc belong to camera group 103h. In this case, camera 102gc is placed closer to cameras 102ga and 102gb than to cameras 102ha, 102hb, and 102hc. The reason is as follows. Assume that camera 102gc is placed closer to camera 102ha. This is because the distance between camera 102gc and the cameras 102 in camera group 103f, which is located on the opposite side of camera group 103h, becomes wider, resulting in a direction in which the image quality of the virtual viewpoint image deteriorates.

Next, an example of capturing images of the entire capture target area 503 using multiple cameras 102 belonging to a camera group 103 will be described with reference to Figures 10 to 12.

Figure 10 shows an example of a camera group 103i in which the entire area of the shooting target area 503 is shot by three cameras 102ia, 102ib, and 102ic. The shooting range of camera 102ia is 901ia. The shooting range of camera 102ib is 901ib. The shooting range of camera 102ic is 901ic. The shooting target area corresponds to an area in which the resolution of the reference subject in the shot image exceeds the lower limit resolution. As shown in Figure 10, when these shooting ranges 901ia, 901ib, and 901ic are combined, the entire area of the shooting target area 503 is shot. For example, camera 102ib is placed in the middle, camera 102ia is placed to the right, and camera 102ic is placed to the left, and the three cameras 102 are placed in a fan shape. As a result, the shooting ranges 901ia, 901ib, and 901ic also spread out in a fan shape.

Figure 10 is a diagram explaining the shooting ranges 901ia to 901ic of the cameras 102ia to 102ic in the in-plane direction of the shooting target area 503. However, the shooting target area may be a three-dimensional space including a predetermined height of the field 501. In that case, the cameras 102 must be arranged so as to shoot the shooting target area including the predetermined height direction of the field 501. Note that since the positions of the optical centers of the three cameras 102 do not match, for example, the shooting ranges 901ic and 901ib partially overlap. By arranging the three cameras 102 as close as possible, the overlap of the shooting ranges 901 can be reduced, enabling a more efficient arrangement. In other words, the number of cameras 102 required for the camera group 103 can be reduced.

Here, the intersection of the optical axis of camera 102 and field 501 of stadium 702 is defined as the gaze point 1001 of camera 102. In FIG. 10, the gaze point of camera 102ia is 1001ia. The gaze point of camera 102ib is 1001ib. The gaze point of camera 102ic is 1001ic. As shown in FIG. 10, the positions of the gaze points 1001 are all different. In other words, the gaze points of the multiple cameras 102 belonging to a camera group 103 are all different. However, one camera group 103 may include cameras 102 with the same gaze point.

In the example of the arrangement of the cameras 102 in FIG. 10, the optical axes of the cameras 102ia to 102ic are oriented in different directions. In addition, the projected lines connecting each of the cameras 102ia to 102ic with their respective gaze points onto the field 501 do not intersect with each other.

Figure 11 is a diagram showing an example of the arrangement of cameras 102 belonging to another camera group 103. Camera group 103g includes six cameras 102ga, 102gb, 102gc, 102gd, 102ge, and 102gf. The entire area of the shooting target area 503 is shot by these cameras 102ga, 102gb, 102gc, 102gd, 102ge, and 102gf.

Figure 11 (a) is a top view of the imaging target area 503, and shows the arrangement of 102ga, 102gb, 102gc, 102gd, 102ge, and 102gf in the in-plane direction of the imaging target area 503. For camera 102ga, the imaging range is 901ga and the gaze point is 1101ga. The same is true for cameras 102gb to 102gf. The six cameras 102 are further divided into two subgroups. The first subgroup includes cameras 102ga, 102gb, and 102gc, which are positioned so as to capture the near side of the imaging target area 503 (the side closer to the arrangement area 701g of camera group 103g) when viewed from the arrangement area 701. The second subgroup includes cameras 102gd, 102ge, and 102gf, which are positioned so as to capture the back side of the imaging target area 503 (the side farther from the imaging area 701g of camera group 103g) when viewed from the placement area 701. As shown in FIG. 11(a), the cameras 102 in each subgroup are positioned so that the imaging range 901 spreads out in a fan shape.

11B is a diagram showing the arrangement of the cameras 102 in the height direction of the shooting target area 503, as viewed from the side. As shown in FIG. 11B, the cameras 102ga, 102gb, and 102gc that shoot the front side of the shooting target area 503 and the cameras 102gd, 102ge, and 102gf that shoot the back side of the shooting target area 503 are arranged at different heights. For example, the camera 102gb that shoots the front side of the shooting target area 503 has a larger depression angle than the camera 102ge that shoots the back side. Therefore, by arranging the camera 102gb that shoots the front side of the shooting target area 503 on the lower side, it is possible to arrange them closer together without physical interference. In other words, the multiple cameras 102 belonging to the camera group 103 may be divided into subgroups, and the heights of the subgroups may be changed.

Figure 12 is a diagram showing another example of the arrangement of cameras 102 belonging to camera group 103g. Camera group 103g includes six cameras 102ga, 102gb, 102gc, 102gd, 102ge, and 102gf. The entire area of the shooting target area 503 is shot by these cameras 102ga, 102gb, 102gc, 102gd, 102ge, and 102gf. In this example, the subgroups are arranged so that their positions differ. For example, the arrangement area 701g is set to include two pillars. Then, a different subgroup is assigned to each of the two pillars. For example, cameras 102ga, 102gb, and 102gc belonging to the first subgroup are arranged on the pillar on the left as viewed from the shooting target area 503. Cameras 102gd, 102ge, and 102gf belonging to the second subgroup are placed on the pillar to the right as viewed from the target area 503. In other words, the multiple cameras 102 belonging to the camera group 103 may be divided into subgroups, and the cameras 102 may be placed at different positions for each subgroup.

If the lower limit resolution is different, the number of cameras 102 for each camera group 103 will be different. If the lower limit resolution is increased, the focal length of the cameras 102 must be increased. If the focal length is increased, the shooting range 901 of the cameras 102 will be narrowed. Therefore, the number of cameras 102 required to shoot the entire shooting target area 503 will increase. Conversely, if the lower limit resolution is decreased, the focal length of the cameras 102 can be shortened. If the focal length is shortened, the shooting range 901 of the cameras 102 will be widened. Therefore, the number of cameras 102 required to shoot the entire shooting target area 503 will decrease.

In step S406, the placement determination unit 306 adds up the number of cameras 102 determined for each camera group 103 determined in S405, and determines the number of cameras 102 required for the multi-camera system 101.

In step S407, the placement determination unit 306 determines whether there is a problem with the number of cameras calculated in step S406. Specifically, the placement determination unit 306 determines that there is a problem if the number of cameras is more or less than a predetermined number. If it is determined that there is no problem (YES), the processing ends. If it is determined that there is a problem (NO), the processing proceeds to step S408. Note that the predetermined number may be set based on a budget, or may be set based on the processing capacity of the virtual viewpoint image generation device 106, the transmission bandwidth capacity when transmitting the captured images, etc.

In step S408, the placement determination unit 306 determines which placement condition to change. The placement conditions that can be changed are the shooting target area 503, the minimum resolution, or the number of camera groups 103. The placement condition to be changed may be automatically determined by the placement determination unit 306, or may be directly input by the operator via the operation unit 206. Even if the placement condition is automatically determined by the placement determination unit 306, the operator may set a priority for the placement condition to be changed in advance, or may specify in advance which placement condition to change.

In step S409, the placement determination unit 306 determines whether the placement condition to be changed is the shooting target area 503. If it is the shooting target area 503 (YES), the process proceeds to step S410. If not (NO), the process proceeds to step S411.

In step S410, the imaging target area acquisition unit 304 changes the imaging target area 503. If the reason for determining that there is a problem in step S407 is that there is a "large number of cameras", the imaging target area 503 is changed to be narrower. Conversely, if the reason is that there is a "small number of cameras", the imaging target area 503 is changed to be wider. When the imaging target area acquisition unit 304 changes the imaging target area 503, the process proceeds to step S405, where the placement determination unit 306 re-determines the number and placement of the cameras 102 based on the changed imaging target area 503. The imaging target area acquisition unit 304 may acquire the imaging target area from the operator via the operation unit 206.

In step S411, the placement determination unit 306 determines whether the placement condition to be changed is the lower limit resolution. If it is the lower limit resolution (YES), the process proceeds to step S412. If not (NO), the process proceeds to step S413.

In step S412, the tolerance acquisition unit 303 changes the lower limit resolution. If the reason for determining that there is a problem in step S407 is that there is a "large number of cameras," the lower limit resolution is changed to a smaller value. Conversely, if the reason is that there is a "small number of cameras," the lower limit resolution is changed to a larger value. After the tolerance acquisition unit 303 has changed the lower limit resolution, the process proceeds to step S405, where the placement determination unit 306 re-determines the number and placement of the cameras 102 based on the changed lower limit resolution. The tolerance acquisition unit 303 may also acquire the lower limit resolution from the operator via the operation unit 206.

In step S413, the camera group number acquisition unit 305 changes the number of camera groups 103. If the reason for determining that there is a problem in step S407 is that there is a "large number of cameras", the number of camera groups 103 is changed to decrease. Conversely, if the reason is that there is a "small number of cameras", the number of camera groups 103 is changed to increase. After the number of camera groups 103 is changed by the camera group number acquisition unit 305, the process proceeds to step S404, where the placement determination unit 306 re-determines the placement area 701 of the camera groups 103. Then, the process proceeds to step S405, where the placement determination unit 306 re-determines the number and placement of the cameras 102. The camera group number acquisition unit 305 may acquire the number of camera groups from the operator via the operation unit 206.

The effect of the arrangement determined by the arrangement determination device 108 will be described with reference to FIG. 13. In each camera group 103, the cameras 102 belonging to the camera group 103 are arranged so that the entire area of the shooting target area 503 is shot. That is, an object at any position in the shooting target area 503 is shot by at least one camera 102 in each camera group 103. Consider an object at a position 1301. In camera group 103a, camera 102aa shoots the object at the position 1301. In camera group 103b, camera 102ba shoots the object at the position 1301. The same applies below. In other words, an object at any position in the shooting target area 503 is shot by the same number of cameras 102 as the number of camera groups 103. In addition, the arrangement areas 701 of each camera group 103 are determined to be as evenly spaced as possible. That is, it is possible to generate a virtual viewpoint image with good image quality for an object at any position in the shooting target area 503.

<Embodiment 2>
In this embodiment, an example of an arrangement determination device 1600 different from that of the first embodiment will be described. Fig. 16 shows an example of the functional configuration of the arrangement determination device 1600 in this embodiment. As shown in Fig. 16, the arrangement determination device 1600 has a camera number acquisition unit 1601 instead of the allowable value acquisition unit 303 in Fig. 3. Also, the arrangement determination device 1600 has an arrangement determination unit 1602 instead of the arrangement determination unit 306 in Fig. 3. Note that the hardware configuration of the arrangement determination device 108 is omitted since it is the same as that in Fig. 2.

The camera number acquisition unit 1601 acquires the number of cameras 102 in the multi-camera system 101 directly from the operator via the operation unit 206. For example, the number of cameras 102 may be set based on the processing performance of the virtual viewpoint image generation device 106. The higher the processing performance of the virtual viewpoint image generation device 106, the more images it can process. In other words, the higher the processing performance of the virtual viewpoint image generation device 106, the higher the number of cameras 102 can be set to. The number of cameras 102 may also be set based on the budget. Increasing the number of cameras 102 increases the cost required for placement. In other words, the larger the budget, the higher the number of cameras 102 can be set to.

The camera number acquisition unit 1601 may also have the following configuration. That is, the camera number acquisition unit 1601 associates the number of cameras 102 with the scale of the event and the density of subjects, which are one of the event information, and stores a table showing the correspondence. Then, when the camera number acquisition unit 1601 receives input of information identifying an event from an operator via the operation unit 206, it may refer to the event information stored in the event information storage unit 301 and the table to acquire the number of cameras 102. For example, when the scale of the event is large, the number of cameras 102 is determined to be greater than when the scale of the event is small. Also, when the density of subjects is large, the number of cameras 102 is determined to be greater than when the density of subjects is small.

The placement determination unit 1602 determines the placement of the cameras 102 based on the number of cameras 102, the shooting target area 503, the number of camera groups, and the venue information. The number of cameras 102 is acquired by the camera number acquisition unit 1601, the shooting target area 503 is acquired by the shooting target area acquisition unit 304, and the number of camera groups is acquired by the camera group number acquisition unit 305.

Figure 14 is a flowchart showing an example of the processing procedure of the placement determination device 108. Steps S402 to S404 are the same as those in Figure 4, so their explanation will be omitted.

In step S1401, the camera number acquisition unit 1601 acquires the number of cameras 102. For example, the number of cameras 102 may be set based on the processing performance of the virtual viewpoint image generation device 106. The higher the processing performance of the virtual viewpoint image generation device 106, the more images can be processed. In other words, the higher the processing performance of the virtual viewpoint image generation device 106, the greater the number of cameras 102 can be. For example, the number of cameras 102 may be set based on a budget. Increasing the number of cameras 102 increases the cost required for placement. In other words, the larger the budget, the greater the number of cameras 102 can be.

In step S1402, the placement determination unit 1602 assigns the number of cameras 102 acquired in step S1401 to the number of camera groups 103 acquired in step S403. The number of cameras 102 required for the camera group 103 varies depending on the relationship between the shooting target area 503 acquired in step S402 and the placement area 701. For example, as shown in FIG. 15, the shooting target area 503 as seen from the placement area 701 is represented by angle 1501. The angle 1501 of the shooting target area 503 as seen from the placement area 701a is 1501a. Then, the cameras 102 are assigned to the camera groups 103 according to the angle 1501. In other words, more cameras 102 are assigned to camera groups 103 in which the angle 1501 of the shooting target area 503 as seen from the placement area 701 is large.

In step S1403, the placement determination unit 1602 determines the placement of the cameras 102 for each camera group 103 based on the number of cameras 102 for each camera group 103 assigned in step S1402 and the venue information. The specific placement of the cameras 102 is determined so that the entire area of the shooting target area 503 is shot. Furthermore, the cameras 102 are placed so that the resolution is as high as possible throughout the entire shooting target area 503.

In step S1404, the placement determination unit 1602 calculates the resolution for the entire shooting target area 503 for each camera group 103. In areas that overlap with the shooting range of a camera 102, the subject is shot from multiple cameras 102. In that case, the value of the camera 102 with the highest resolution is set as the resolution.

In step S1405, the placement determination unit 1602 determines whether there is a problem with the resolution calculated in step S1404. For example, it is determined that there is a problem if there is an area in the shooting target area 503 whose resolution is smaller than a predetermined value. If it is determined that there is no problem (YES), the process ends. If it is determined that there is a problem (NO), the process proceeds to step S1406.

In step S1406, the placement determination unit 1602 determines which placement condition to change. The placement conditions that can be changed are either the shooting target area 503 or the number of camera groups 103.

In step S1407, the placement determination unit 1602 determines whether the condition to be changed is the subject area 503. If the condition to be changed is the subject area 503 (YES), the process proceeds to step S1408. If the condition to be changed is not the subject area 503 (NO), the process proceeds to step S1409.

In step S1408, the imaging target area acquisition unit 304 changes the imaging target area 503. Specifically, the imaging target area acquisition unit 304 changes the imaging target area 503 to be narrower so that the calculated resolution is higher. After the imaging target area acquisition unit 304 changes the imaging target area 503, the process proceeds to step S404, where the placement determination unit 306 re-determines the placement area 701 of the camera group 103. Then, the process proceeds to steps S1402 and S1403, where the placement determination unit 306 re-determines the number and placement of the cameras 102. Note that the imaging target area acquisition unit 304 may acquire the imaging target area from the operator via the operation unit 206.

In step S1409, the camera group number acquisition unit 305 changes the number of camera groups 103. Specifically, the camera group number acquisition unit 305 changes the number of camera groups 103 to reduce the number so as to increase the calculated resolution. This is because reducing the number of camera groups 103 increases the number of cameras 102 assigned to the camera groups 103, resulting in higher resolution. After the camera group number acquisition unit 305 changes the number of camera groups 103, the process proceeds to step S404, where the placement determination unit 306 re-determines the placement area 701 of the camera group 103. Then, the process proceeds to steps S1402 and S1403, where the placement determination unit 306 re-determines the number and placement of the cameras 102. The camera group number acquisition unit 305 may acquire the number of camera groups from the operator via the operation unit 206.

In this embodiment, the same effect as in the first embodiment can be obtained. That is, a subject at any position in the shooting target area 503 is captured by the same number of cameras 102 as the number of camera groups 103. In addition, the placement areas 701 of the camera groups 103 are determined to be as equally spaced as possible. In other words, it is possible to generate a high-quality virtual viewpoint image of a subject at any position in the shooting target area 503.

<Embodiment 3>
In this embodiment, an example of the arrangement determination device 1800 different from that of the first embodiment will be described. FIG. 18 shows an example of the functional configuration of the arrangement determination device 1800 in this embodiment. As shown in FIG. 18, the arrangement determination device 1800 has an event information storage unit 1801, a venue information storage unit 1802, a tolerance acquisition unit 1803, a shooting target area acquisition unit 1804, a camera group number acquisition unit 1805, and an arrangement determination unit 1806. Furthermore, the arrangement determination device 1800 has a type acquisition unit 1807, an area setting unit 1808, a level information setting unit 1809, and a group number setting unit 1810. However, the type acquisition unit 1807, the area setting unit 1808, the level information setting unit 1809, and the group number setting unit 1810 are functional units used to generate event information, and the arrangement determination device does not need to have these functional units. In other words, if event information generated by the type acquisition unit 1807, area setting unit 1808, level information setting unit 1809, and number of groups setting unit 1810 is stored in the event information storage unit 1801, the camera placement can be determined from the event information. Note that the hardware configuration of the placement determination device 1800 is omitted since it is the same as that in Fig. 2. The following description will focus on the differences from the first embodiment.

In this embodiment, multiple spaces are defined for a space to be photographed by multiple cameras (e.g., space 504 in FIG. 5(b)), and each of the multiple defined spaces is set as a photographing target area. Then, the layout determination device 1800 in this embodiment determines the number and layout of camera groups, and the number and layout of cameras in each camera group, for the multiple photographing target areas. However, this embodiment does not need to be applied to all of the multiple photographing target areas, and this embodiment may be applied to one of them.

All of the multiple spaces defined are included in the space that is the subject of image capture by the multiple cameras (space 504 in FIG. 5(b)). The multiple spaces defined may be defined mutually exclusive. In other words, the multiple spaces may be defined so as to be composed of multiple spaces that divide space 504 in FIG. 5(b). Of the multiple spaces defined, any two of the spaces may overlap, and one space may be contained within the other space.

The type acquisition unit 1807 acquires the type of the main foreground (subject) appearing in the event and its range of movement. The type acquisition unit 1807 acquires information by inputting information on the type of subject and its range of movement by the operator via the operation unit 206. An example of this information is shown in FIG. 19(a). Here, rugby is assumed as the event, and a person (player) and a ball are input as the type of the main subject, and an example is shown in which the person is input separately as a still person and a moving person. Furthermore, the range in which each subject moves is input corresponding to the type of subject. Here, the range of movement indicates the range in which a person or a ball may exist during a rugby game. For example, the range of movement of a still person indicates the range in which a still person may exist. Therefore, the range of movement of a still person is approximately the same as the length of the length and width of the field. The height direction is, for example, the same as the height of a player (for example, 2 m). On the other hand, the range of movement of a moving person is wider than the range of movement of a still person. For example, in terms of height, in rugby, a player may lift another player during a lineout throw-in, so the range of movement of a moving person is wider than that of a stationary person. Specifically, the height is about 5m. The range of movement of the ball is even wider. For example, in rugby, the kicked ball goes up high, so the height of the ball's range of movement is set to 20m. Also, in the vertical and horizontal directions, that is, in the plane of the field, a moving person or the ball has a wider range of movement than a stationary person.

The information acquired by the type acquisition unit 1807 is output to the area setting unit 1808 and also to the event information storage unit 1801, where it is stored. Specifically, the event information storage unit 1801 stores the table shown in FIG. 19(a). However, the event information storage unit 1801 may store the information output by the type acquisition unit 1807 in a format other than the table in FIG. 19(a).

19(a), the operator inputs the type of event and other information that can identify the event, information on the size of the field, information on the density of the subjects, and information on the use of the virtual viewpoint image, as in the first embodiment. However, if the placement determination device 1800 is only used for specific events, input of the type of event and other information that can identify the event can be omitted.

The area setting unit 1808 sets multiple shooting target areas based on the information indicating the type of subject and its movement range acquired by the type acquisition unit 1807. However, among the multiple shooting target areas, any two or more shooting target areas may partially overlap. Figures 20(a) and (b) show the relationship between the shooting target areas in an example in which three shooting target areas are set. Figure 20(a) is a schematic diagram of the field viewed from above, showing the relationship between the vertical and horizontal sizes of the three shooting target areas. Also, Figure 20(b) is a diagram of the field viewed from the side, showing the relationship between the heights of the three shooting target areas. Here, the first shooting target area 2001 corresponds to the movement range of a stationary person, the second shooting target area 2002 corresponds to the movement range of a moving person, and the third shooting target area 2003 corresponds to the movement range of the ball. The first shooting target area 2001, the second shooting target area 2002, and the third shooting target area 2003 have wider ranges in this order. In addition, the first photographed area 2001 is included in the second photographed area 2002, and the second photographed area 2002 is included in the third photographed area 2003. However, only a part of the first photographed area 2001, not all of it, may be included in the second photographed area 2002. Only a part of the second photographed area 2002, not all of it, may be included in the third photographed area 2003.

Information indicating the multiple shooting target areas set by the area setting unit 1808 is output to the level information setting unit 1809, and is also output to the event information storage unit 1801, and is stored by the event information storage unit 1801.

The area setting unit 1808 may set the shooting target area so that the same shooting target area corresponds to the movement range of a stationary person and a moving person. Specifically, the area setting unit 1808 may not set the first shooting target area 2001 in Figs. 20(a) and (b), but may set the remaining second shooting target area 2002 and third shooting target area 2003.

The level information setting unit 1809 sets the required level of quality of the three-dimensional shape data and the required level of the minimum resolution for each of the multiple shooting target areas set by the area setting unit 1808. The level information setting unit 1809 accepts input from the operator via the operation unit 206 and sets the required level. An example of the required level of quality of the three-dimensional shape data and the required level of the minimum resolution corresponding to each shooting target area is shown in FIG. 19(b). Here, the required level of quality of the three-dimensional shape data and the required level of the minimum resolution are defined in three stages: "high", "medium", and "low". Note that the required level of the minimum resolution may be defined by the number of pixels representing the minimum resolution, as shown in the first embodiment.

The main subject of the first shooting target area 2001 is a still person. For a still person, the image quality of the virtual viewpoint image can be improved by displaying the still person in high image quality in the virtual viewpoint image. Therefore, the level information setting unit 1809 sets the required level of the quality of the three-dimensional shape data and the required level of the minimum resolution to "high" for the first shooting target area 2001 corresponding to the movement range of the still person. Note that a required level of the minimum resolution of "high" means a level at which at least the facial features and contours are clearly visible. As a specific numerical example, the number of pixels for a height of 2 m in the shooting image is set to 400 pixels. This is equivalent to requiring a resolution of at least 0.5 cm/pixel.

The main subject of the second shooting target area 2002 is a moving person. Unlike a still person, a moving person is moving, and therefore the image quality of the moving person can be tolerated even if it is lower than the image quality of a still person. Therefore, the required level of quality of the three-dimensional shape data for the second shooting target area 2002 and the minimum resolution level are set to "medium." As a specific numerical example of the minimum resolution level, the number of pixels for a height of 2 m in the shooting image is set to 200 pixels.

The main subject of the third shooting target area 2003 is the ball. In rugby, the ball can move faster than a person and is smaller than a person, so its image quality can be tolerated even if it is lower than the image quality of a moving person.

In FIG. 19(b), the required level of quality of the 3D shape data for one shooting target area and the required level of the minimum resolution are the same, but depending on the event, the required level of quality of the 3D shape data and the required level of the minimum resolution do not have to be the same. For example, the required level of quality of the 3D shape data for one shooting target area may be "high," and the required level of the minimum resolution for that shooting target area may be "low."

The information set by the level information setting unit 1809 is output to the number of groups setting unit 1810 and the event information storage unit 1801, and is stored by the event information storage unit 1801. Specifically, the event information storage unit 1801 stores the table shown in FIG. 19(b). However, the event information storage unit 1801 may store the information output by the type acquisition unit 1807 in a format other than the table in FIG. 19(b).

The number of groups setting unit 1810 sets the number of camera groups 103 based on the required level of quality of the three-dimensional shape data set by the level information setting unit 1809. The greater the number of camera groups, the higher the accuracy of the three-dimensional shape data. Therefore, the higher the required level, the greater the number of camera groups. Each shooting target area and the number of camera groups are shown in FIG. 19(c). Here, an example is shown in which the number of camera groups is set to "24" if the required level of quality of the three-dimensional shape data is "high", "12" if it is "medium", and "8" if it is "low". The information set by the number of groups setting unit 1810 is also output to the event information storage unit 1801 and stored by the event information storage unit 1801. Specifically, the event information storage unit 1801 stores the table shown in FIG. 19(c). However, the event information storage unit 1801 may store the information output by the type acquisition unit 1807 in a format other than the table in FIG. 19(c).

The group number setting unit 1810 may set the number of camera groups uniquely based on the required level of quality of the three-dimensional shape data, but may also set it based on the density of the subjects. In the example of rugby, the density is high in the first shooting target area 2001 because plays such as scrums and mauls occur there, but the density is low in the second shooting target area 2002 and the third shooting target area 2003. Taking this into consideration, the group number setting unit 1810 sets the number of camera groups. In other words, the group number setting unit 1810 sets the number of camera groups so that the higher the density, the greater the number of camera groups.

The event information storage unit 1801 stores information about events involving the subject of photography. In addition to the information shown in embodiment 1, the event information may include information set for each event indicating the type of subject, information indicating the photography area, and information on the density of subjects in each photography area.

The venue information storage unit 1802 stores information about the venue in which the multiple cameras 102 are arranged. The venue information is the same as the information shown in embodiment 1.

When the camera 102 captures a predetermined reference subject, the tolerance acquisition unit 1803 sets a lower limit of the resolution, which is the size of the subject in the image. Specifically, the tolerance acquisition unit 1803 sets a lower limit for each of the capture target areas based on information indicating the correspondence between multiple capture target areas set for each event and the required level of the minimum resolution, which is one of the event information. The information indicating the correspondence between multiple capture target areas set for each event and the required level of the minimum resolution may be stored in the event information storage unit 1801 as a correspondence table as shown in FIG. 19(b) or may be stored in an external storage device. In this way, when the tolerance acquisition unit 1803 receives input of information that can identify an event from the operator via the operation unit 206, it sets a lower limit of the resolution for multiple capture target areas set for that event.

The tolerance acquisition unit 1803 may also acquire information such as the numerical value of the lower limit resolution directly from the operator via the operation unit 206.

Based on the event information, the shooting target area acquisition unit 1804 determines multiple shooting target areas corresponding to the event. Specifically, the shooting target area acquisition unit 1804 may be configured as follows. That is, first, the shooting target area acquisition unit 1804 stores a table in which at least a part of the event information is associated with information representing multiple shooting target areas. The at least a part of the event information is information indicating the type of subject, information on the movement range of the subject, information on the size of the field, etc. Then, when the shooting target area acquisition unit 1804 receives input of information that can identify an event from the operator via the operation unit 206, it refers to the event information stored in the event information storage unit 1801 and the table, and determines multiple shooting target areas. Note that the referenced table may be, but is not limited to, information on the type of subject, information on the movement range of the subject, and information on multiple shooting target areas that are associated with each other. Event information other than the above may be associated with information on the shooting target areas. Also, the referenced table may be information that can identify an event, such as the name of the event, and multiple shooting target areas that are associated with each other. Furthermore, the referenced table may be stored in the event information storage unit 1801, or may be stored in an external device.

The imaging target area acquisition unit 1804 may also acquire information about the imaging target area directly from the operator via the operation unit 206.

The camera group number acquisition unit 1805 may set the camera groups by referring to a table in which the event information and the number of camera groups 103 are associated. That is, when the camera group number acquisition unit 305 accepts an input of information that can identify an event via the operation unit 206, it refers to the event information stored in the event information storage unit 1801 and the table, and acquires the number of camera groups 103 for each shooting target area. An example of the table to be referenced is shown in FIG. 19(c). Note that the table to be referenced may be one in which information on the shooting target area and information on the number of camera groups 103 are associated, but is not limited to this. Also, the table to be referenced may be stored in the event information storage unit 1801, or may be stored in an external device.

The camera group number acquisition unit 1805 may also acquire information on the number of camera groups 103 directly from the operator via the operation unit 206.

The placement determination unit 1806 determines the placement of the cameras 102 based on the lower limit resolution, multiple target areas, the number of camera groups for each target area, and venue information. The lower limit resolution is acquired by the tolerance acquisition unit 1803, the multiple target areas are acquired by the target area acquisition unit 1804, and the number of camera groups is acquired by the camera group number acquisition unit 1805. The venue information is stored in the venue information storage unit 302. The placement determination unit 306 acquires the necessary venue information from the venue information storage unit 302 based on information for identifying the venue input via the operation unit 206. For example, the acquired venue information includes information on structures within the venue where the cameras 102 can be placed and their positions, information on the number of cameras 102 that can be placed, etc.

First, the placement determination unit 1806 determines the placement positions of the camera groups 103 for each of the multiple shooting target areas based on the number of camera groups and the venue information, as in the first embodiment. Next, the placement determination unit 306 determines the placement of the multiple cameras 102 belonging to each camera group 103. At that time, the placement determination unit 306 determines the placement of the multiple cameras 102 belonging to each camera group 103 so that the entire area of each shooting target area is photographed by the multiple cameras 102 belonging to the camera group 103. In other words, the placement of the multiple cameras 102 belonging to the camera group 103 of the first shooting target area 2001 is determined so that the entire area of the first shooting target area 2001 is photographed by the multiple cameras 102 belonging to this camera group 103. The placements of the multiple cameras 102 belonging to the camera group 103 of the second shooting target area 2002 and the multiple cameras 102 belonging to the camera group 103 of the third shooting target area 2003 are also determined in the same manner.

Furthermore, the placement determination unit 1806 determines the placement of the cameras 102 belonging to the camera group 103 so that the resolution exceeds the lower limit resolution in the entire area of each of the shooting target areas. Note that, although the placement of the cameras 102 is determined so that the above conditions regarding the shooting target area and the resolution conditions are satisfied in all camera groups 103 corresponding to each shooting target area, this is not limited to this. For example, the placement of the cameras 102 may be determined so that only the cameras 102 belonging to a predetermined number of camera groups 103 satisfy the above conditions. Also, the placement of the cameras 102 belonging to the camera group 103 may be determined so that the camera group 103 at a predetermined position or the camera group 103 in a predetermined relationship satisfies the above conditions. In other words, the cameras 102 belonging to some camera groups 103 may be placed so that the cameras 102 capture not the entire area but a part of each shooting target area. Furthermore, in multiple cameras 102 belonging to some camera groups 103, the resolution does not have to exceed the lower limit resolution in some areas of each shooting target area.

Figure 21 is a flowchart showing an example of the processing procedure of the placement determination device 1800. The processing performed by the placement determination device 1800 will be explained using Figure 21.

First, the placement determination device 1800 receives a placement determination instruction from the operator via the operation unit 206. Then, in step S2101, the shooting target area acquisition unit 1804 determines and acquires multiple shooting target areas. Specifically, the multiple shooting target areas are determined based on information capable of identifying an event input by the operator via the operation unit 206 and a pre-stored table. As the multiple shooting target areas, a first shooting target area 2001, a second shooting target area 2002, and a third shooting target area 2003 are set as shown in FIG. 20.

Next, in step S2102, the tolerance acquisition unit 1803 acquires the required quality level and the minimum resolution level of the three-dimensional shape data. The tolerance acquisition unit 1803 determines and acquires the required quality level and the minimum resolution level of the three-dimensional shape data for each imaging target area based on information capable of identifying the event input by the operator via the operation unit 206 and a pre-stored table.

Next, in step S2103, the camera group number acquisition unit 1805 determines and acquires the number of camera groups 103 for each shooting target area. The number of camera groups 103 is determined according to the required level of quality of the three-dimensional shape data generated by the virtual viewpoint image generation device 106. The number of directions required to generate this three-dimensional shape data varies depending on the required level of quality of the three-dimensional shape data. In other words, when the required level of quality of the three-dimensional shape data is high, that is, when the accuracy of the three-dimensional shape data is to be high, shooting from many directions is required, so the number of camera groups 103 increases. On the other hand, when the required level of quality of the three-dimensional shape data is low, the accuracy of the three-dimensional shape data may be low, so the number of shooting directions may be reduced, and as a result, the number of camera groups 103 can be reduced.

The camera group number acquisition unit 1805 can determine the number of camera groups by referring to a table that associates the required quality level of the three-dimensional shape data with the number of required directions (i.e., the number of camera groups). Since the required quality level of the three-dimensional shape data corresponds to the area to be photographed, here, the number of camera groups is determined by referring to a table ( FIG. 19(c) ) that associates the area to be photographed with the number of camera groups, rather than the required quality level of the three-dimensional shape data.

As an example, let us assume that the numbers of camera groups 103 in the first imaging target area 2001, the second imaging target area 2002, and the third imaging target area 2003 are determined to be 30, 24, and 10, respectively. The placement areas of the camera groups 103 may also be determined in this process, but in this embodiment, an example is shown in which they are determined in the next step S2104.

In step S2104, the placement determination unit 1806 determines the placement area of the camera group 103 for each shooting target area based on the number of camera groups and the venue information. Note that this process is equivalent to performing the same process as S404 in FIG. 4 for each of the multiple shooting target areas, so a description of this process is omitted.

In step S2105, the placement determination unit 1806 determines the number and placement of two or more cameras 102 belonging to the camera group 103 for each camera group 103 in each shooting target area based on the placement area of the camera group 103 and the minimum required resolution level. This process is equivalent to performing the same process as S405 in FIG. 4 for each of the multiple shooting target areas, so a description thereof will be omitted. The placement of the cameras 102 determined by the placement determination unit 1806 is also the same as in embodiment 1.

In step S2106, the placement determination unit 1806 adds up the number of cameras 102 determined for each camera group 103 in each shooting target area in S2105, and determines the number of cameras 102 required for the multi-camera system 101.

In step S2107, the placement determination unit 1806 determines whether there is a problem with the number of cameras calculated in step S2106. Note that step S2107 is the same process as S407 in FIG. 4, so a detailed description will be omitted. If it is determined that there is no problem (YES), the process ends. If it is determined that there is a problem (NO), the process proceeds to step S2108.

In step S2108, the placement determination unit 1806 determines which placement condition to change. The placement conditions that can be changed are the subject area, the minimum resolution, and the number of camera groups 103. Step S2108 is the same process as S408 in FIG. 4, so a detailed description will be omitted.

In step S2109, the placement determination unit 1806 determines whether the placement condition to be changed is the area to be photographed. If it is the area to be photographed (YES), the process proceeds to step S2110. If not (NO), the process proceeds to step S2111.

In step S2110, the imaging target area acquisition unit 1804 changes the imaging target area. Note that one or more of the multiple imaging target areas are changed. In other words, it is not necessary to change all of the imaging target areas. If the reason for determining that there is a problem in step S2107 is that there is a "large number of cameras", one or more of the multiple imaging target areas are changed to be narrower. Conversely, if there is a "small number of cameras", one or more of the multiple imaging target areas are changed to be wider. When the imaging target area acquisition unit 1804 changes the imaging target area, the process proceeds to step S2105, and the arrangement determination unit 1806 re-determines the number and arrangement of the cameras 102 based on the multiple imaging target areas including the changed imaging target area. Note that the imaging target area acquisition unit 1804 may acquire the imaging target area from the operator via the operation unit 206.

In step S2111, the placement determination unit 1806 determines whether the placement condition to be changed is the lower limit resolution. If it is the lower limit resolution (YES), the process proceeds to step S2112. If not (NO), the process proceeds to step S2113.

In step S2112, the tolerance acquisition unit 1803 changes the minimum resolution requirement level. If the reason for determining that there is a problem is that there is a "large number of cameras", the minimum resolution requirement level of one or more shooting target areas is changed to be smaller. Conversely, if the reason for determining that there is a problem is that there is a "small number of cameras", the minimum resolution requirement level of one or more shooting target areas is changed to be larger. After the minimum resolution is changed by the tolerance acquisition unit 1803, the process proceeds to step S2105, where the placement determination unit 1806 re-determines the number and placement of the cameras 102. The tolerance acquisition unit 1803 may acquire the minimum resolution requirement level from the operator via the operation unit 206.

In step S2113, the camera group number acquisition unit 1805 changes the number of camera groups 103. If the reason for determining that there is a problem in step S2107 is that there is a "large number of cameras", the number of camera groups 103 in one or more shooting target areas is changed to decrease. Conversely, if the reason for determining that there is a problem is that there is a "small number of cameras", the number of camera groups 103 in one or more shooting target areas is changed to increase. After the number of camera groups 103 is changed by the camera group number acquisition unit 1805, the process proceeds to step S2104, where the placement determination unit 1806 re-determines the placement area of the camera groups 103. Then, the process proceeds to step S2105, where the placement determination unit 1806 re-determines the number and placement of the cameras 102. The camera group number acquisition unit 1805 may acquire the number of camera groups from the operator via the operation unit 206.

According to this embodiment, the same effect as in the first embodiment is achieved. That is, in each camera group 103 of the shooting target area, the multiple cameras 102 belonging to that camera group 103 are arranged so that the entire area of the corresponding shooting target area is shot. Therefore, a subject at any position in the shooting target area is shot by at least one camera 102 in each camera group 103. Then, a subject at any position in the shooting target area is shot by the same number of cameras 102 as the number of camera groups 103. In addition, the placement areas of each camera group 103 are determined so that they are as evenly spaced as possible. That is, it is possible to generate a virtual viewpoint image with good image quality for a subject at any position in the shooting target area.

In addition, in this embodiment, when there are multiple required levels of resolution in the image depending on the type of subject and its movement range, a shooting target area is set for each of the multiple required levels. Then, by determining the number and arrangement of camera groups 103 and the number and arrangement of cameras 102 in the camera group for each shooting target area, an appropriate number of cameras can be arranged in appropriate positions according to the type of subject. For example, when the space 504 in FIG. 5 of the first embodiment is the same as the third shooting target area 2003 in FIG. 20, the number of cameras 102 can be reduced more than in the first embodiment. This is because in the first embodiment, the number of camera groups 103 and the number of cameras 102 are determined uniformly regardless of the type of subject or the movement range. For example, in the first embodiment, the number of cameras 102 is increased because all cameras belonging to the camera group are used to shoot even in areas far from the ground, specifically, in the area of the third shooting target area 2003 that is not included in the second shooting target area 2002. However, since the only subject in that area is the ball, the required level of quality for the 3D shape data can be low, and there is no need to capture the image at high resolution. Therefore, in this embodiment, taking this into consideration, the number of camera groups 103 that capture the image of that area is reduced, and the number of cameras 102 belonging to the camera groups 103 is also reduced.

In addition, in the third embodiment, for example, one or more camera groups may be shared between the first photographing target area 2001 and the second photographing target area. In that case, the number of cameras 102 can be further reduced. In this case, the following processing is performed. That is, the processing of S404 is performed as follows. The layout determination unit 1806 determines the layout of the camera group 103 for the first photographing target area 2001. The layout determination unit 1806 also determines the layout of the camera group 103 for the second photographing target area 2002. Then, it is determined whether there is a camera group 103 determined for the first photographing target area 2001 that is located close to the camera group 103 determined for the second photographing target area 2002. If there is a camera group 103 determined for the second photographing target area 2002 among the camera groups 103 in the first photographing target area 2001, the camera group 103 is removed from the first photographing target area 2001. As explained above, the number and arrangement of the cameras 102 belonging to the camera group 103 are determined according to the minimum resolution requirement level and the quality requirement level of the three-dimensional shape data based on the shooting target area corresponding to the camera group 103. For example, assume that twelve camera groups 701a to 701l as shown in Fig. 7 are determined for the first shooting target area 2001. On the other hand, assume that positions close to the camera groups 701b, 701d, 701f, 701h, 701j, and 701k as shown in Fig. 7 are determined for the second shooting target area 2002. In this case, the camera groups 701b, 701d, 701f, 701h, 701j, and 701k are excluded from the camera groups for the first shooting target area 2001. In other words, camera groups 701a, 701c, 701e, 701g, 701i, and 701l remain as camera groups for the first imaging target area 2001, and the subsequent processing is performed.

In this way, even if a multi-camera system in which one or more camera groups are shared between the first shooting target area 2001 and the second shooting target area 2002 is used, three-dimensional shape data can be appropriately generated. In addition, the camera group of the first shooting target area 2001 and the camera group 103 of the second shooting target area 2002 are arranged alternately, so that the quality level can be reduced in a directional bias when generating three-dimensional shape data. In addition, in the second shooting target area 2002, not only the images captured by the cameras of the camera group 103 of the second shooting target area 2002 but also the images captured by the cameras of the camera group 103 of the first shooting target area 2001 are used to generate three-dimensional shape data of the subject. However, when determining the pixel value of the virtual viewpoint image, only the images captured by the cameras of the camera group 103 of the first shooting target area 2001, which has a high required level of minimum resolution, are used in order to improve image quality. Note that, if the images captured by the cameras of the camera group 103 of the first imaging target area 2001 are used preferentially, images captured by the cameras of the camera group 103 of the second imaging target area 2002 may be used. For example, when the pixel value of a pixel of interest in a virtual viewpoint image is determined by weighted averaging of pixel values at positions corresponding to the pixel of interest in a plurality of captured images, the following is done. That is, the weight of the pixel value at a position corresponding to the pixel of interest in an image captured by a camera of the camera group 103 of the first imaging target area 2001 is made larger than the weight of the pixel value at a position corresponding to the pixel of interest in an image captured by a camera of the camera group 103 of the second imaging target area 2002.

Note that the first imaging target area 2001 and the second imaging target area 2002 are given as examples, but the number of cameras 102 can be reduced by sharing a part of the camera group for any two or more imaging target areas among the multiple imaging target areas.

The type acquisition unit 1807 may acquire information on multiple shooting target areas instead of the type of subject. That is, for example, in FIG. 19(a), information indicating the type of shooting target area (first shooting target area 2001, second shooting target area 2002, third shooting target area 2003) and the length, width, and height that define the shooting target area may be acquired. In addition, although rugby has been used as an example of an event, this embodiment can also be applied to ball games such as soccer and track and field events. For example, in a 100m sprint, which is a track and field event, it is considered that a higher quality virtual viewpoint image is required near the finish line on the track where the runners run than in other places. In that case, various request levels may be set higher for the area near the finish line and lower for the other areas.

<Other embodiments>
The object of the present invention can also be achieved by the following: A storage medium on which program code of software for realizing the functions of the above-mentioned embodiments is recorded is supplied to a system or device. A computer (or a CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

In this case, the program code read from the storage medium itself will realize the functions of the above-mentioned embodiment, and the storage medium storing that program code will constitute the present invention.

Storage media for supplying program code may include, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a non-volatile memory card, a ROM, etc.

This also includes cases where the functions of the above-mentioned embodiments are realized by the following processing. Based on the instructions of the program code read by the computer, an OS (operating system) running on the computer performs some or all of the actual processing.

Furthermore, the functions of the above-mentioned embodiments may be realized by the following process. First, program code read from a storage medium is written to memory on a function expansion board inserted into a computer or on a function expansion unit connected to the computer. Next, based on the instructions of the program code, a CPU or the like on the function expansion board or function expansion unit performs some or all of the actual processing.

304 Shooting target area acquisition unit 306 Arrangement determination unit

Claims (19)

1. An arrangement determination device for determining an arrangement of a plurality of image capture devices used to generate a virtual viewpoint image, comprising:
an area acquisition means for acquiring information representing an area to be photographed by the plurality of photographing devices;
a group acquisition means for acquiring information on a plurality of groups corresponding to a plurality of directions in which the subject area is photographed;
an arrangement determination means for determining an arrangement of the plurality of photographing devices based on the information acquired by the area acquisition means and the information acquired by the group acquisition means, such that the entire area to be photographed is photographed from the first direction by two or more photographing devices included in the plurality of photographing devices and belonging to a first group of the plurality of groups corresponding to a first direction, and the entire area to be photographed is photographed from the second direction by other two or more photographing devices included in the plurality of photographing devices and belonging to a second group of the plurality of groups corresponding to a second direction different from the first direction. a tolerance acquisition means for acquiring a tolerance of the number of pixels representing the size of the reference subject in an image captured by the image capture device when the reference subject is present in the image capture target area,
The arrangement determination device according to claim 1, further comprising: a position determination means for determining an arrangement of the plurality of photographing devices based on the tolerance value acquired by the tolerance acquisition means, such that the entire area to be photographed is photographed from the first direction by two or more photographing devices belonging to the first group among the plurality of photographing devices, the entire area to be photographed is photographed from the second direction by other two or more photographing devices belonging to the second group, and the size of the reference subject in the images captured by each of the plurality of photographing devices exceeds the tolerance value. The arrangement determination device according to claim 2, characterized in that the reference subject is a subject having a predetermined size. The arrangement determination device according to claim 2 or 3, characterized in that the reference subject is a person of a given height who stands upright. The arrangement determination device according to any one of claims 2 to 4, characterized in that the tolerance acquisition means determines the tolerance based on the purpose of the virtual viewpoint image. The arrangement determination device according to claim 5, characterized in that the tolerance is smaller when the virtual viewpoint image is used in a live broadcast than when the virtual viewpoint image is used in an archive distribution. The method further includes: acquiring the number of the photographing devices;
The arrangement determination device according to claim 1, characterized in that the arrangement determination means determines the arrangement of the multiple imaging devices based on the number acquired by the number acquisition means, such that the entire area to be photographed is photographed by two or more of the multiple imaging devices, and the entire area to be photographed is photographed by the other two or more imaging devices. The arrangement determination device according to claim 7, characterized in that the number acquisition means determines the number based on the processing performance of a device that generates a virtual viewpoint image. The arrangement determination device according to any one of claims 1 to 8, further characterized in that the arrangement determination means determines the arrangement of the multiple image capture devices based on information about the venue in which the multiple image capture devices are arranged, so that the entire area to be captured is captured by two or more of the multiple image capture devices, and the entire area to be captured is captured by the other two or more image capture devices. The group acquisition means acquires the number of a plurality of groups,
The placement determination means includes:
determining an arrangement area of the group based on the number of groups acquired by the group acquisition means;
10. The arrangement determination device according to claim 1, further comprising: determining an arrangement of the camera devices belonging to each of the groups for each of the groups. The arrangement determination device according to claim 10, characterized in that the group acquisition means acquires the number of groups based on the type of event. The arrangement determination device according to claim 10 or 11, characterized in that the arrangement determination means determines the arrangement areas of the groups so that the angle formed by the line segment connecting a predetermined position in the shooting target area and each of the arrangement areas of two adjacent groups does not exceed a predetermined value. The arrangement determination device according to any one of claims 10 to 12, characterized in that the arrangement determination means determines the arrangement of the plurality of camera devices so that the distance between the camera devices in a group does not exceed a predetermined distance. The arrangement determination device according to any one of claims 1 to 13, characterized in that the arrangement determination means determines the arrangement of the multiple camera devices such that a camera device, among the two or more camera devices, that is close to the other two or more camera devices is closer to the other camera devices among the two or more camera devices than any of the other two or more camera devices. The arrangement determination device according to any one of claims 1 to 14, characterized in that the imaging target area is at least a part of the space that is the subject of imaging by the multiple imaging devices. The space includes a first imaging target area and a second imaging target area different from the first imaging target area,
the arrangement determination means determines an arrangement of the plurality of photographing devices such that the entire area of the first photographing target area is photographed by two or more first photographing devices among the plurality of photographing devices, and the entire area of the first photographing target area is photographed by two or more second photographing devices among the plurality of photographing devices that are different from the first photographing devices;
The arrangement determination device according to claim 15, characterized in that the arrangement determination means determines the arrangement of the multiple imaging devices such that the entire second imaging target area is captured by two or more third imaging devices among the multiple imaging devices that are different from the first and second imaging devices, and the entire first imaging target area is captured by two or more fourth imaging devices among the multiple imaging devices that are different from the first to third imaging devices. The arrangement determination device according to claim 16, characterized in that the second imaging target area includes the first imaging target area. 1. A method for determining an arrangement of a plurality of image capture devices used to generate a virtual viewpoint image, comprising the steps of:
an area acquisition step of acquiring information representing an imaging target area that is an object to be imaged by the plurality of imaging devices;
a group acquisition step of acquiring information about a plurality of groups corresponding to a plurality of directions in which the subject area is photographed;
and an arrangement determination step of determining an arrangement of the plurality of photographing devices based on information acquired in the area acquisition step and information acquired in the group acquisition step, such that the entire area to be photographed is photographed from the first direction by two or more photographing devices included in the plurality of photographing devices belonging to a first group of the plurality of groups corresponding to a first direction, and the entire area to be photographed is photographed from the second direction by other two or more photographing devices included in the plurality of photographing devices belonging to a second group of the plurality of groups corresponding to a second direction different from the first direction. A program for causing a computer to function as a placement determination device according to any one of claims 1 to 17.

Images