JP7679606B2 - PROCESSING SYSTEM, COMMUNICATION SYSTEM, AND PROCESSING METHOD - Google Patents

Description

This disclosure relates to a processing system, a communication system, and a processing method.

A marketing information provision system has been disclosed that uses multiple cameras to analyze the behavior of people and other objects in wide-area footage and generate behavior trajectory information for the purpose of providing effective promotional information selected based on people's behavior in real time (see Patent Document 1).

However, in the past, while it was possible to capture the movement of an object only within a specific area, it was not known that a single imaging unit could divide the area into multiple areas. This resulted in the problem that it was not possible to detect a specific object when the area was divided into multiple areas.

The invention of claim 1 is a processing system comprising a transmitting means for transmitting image data representing a specified object to a management device that performs image matching; a receiving means for receiving from the management device object identification information for identifying the object assigned as a result of matching by the management device; a display control means for simultaneously displaying multiple specified area images which are images of specified areas among multiple areas contained in a captured image obtained by photographing with a single imaging unit ; and a receiving means for accepting the setting of each of the multiple specified area images as a detectable range in which an object can be detected in the captured image, wherein the detectable range includes a matchable range in which an image of an object can be matched, and the matchable range is a specified area within the detectable range, and the transmitting means is configured to associate first area identification information for identifying a first area among the multiple areas and the object identification information of the object present in the first area and transmit them to the management device.

As described above, according to the present invention, an effect is achieved in that a specific object can be detected even when an area is divided into a plurality of areas by a single imaging unit.

1 is a schematic diagram of a communication system according to an embodiment of the present invention. FIG. 2 is a hardware configuration diagram of a real-time data processing terminal. FIG. 2 is a hardware configuration diagram of an imaging unit. Figure 4(a) shows a hemispherical image (front side) captured by imaging unit 40b, Figure 4(b) shows a hemispherical image (rear side) captured by imaging unit 40b, and Figure 4(c) shows an image represented by equirectangular projection. 1A is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and FIG. 1B is a diagram showing a spherical image. FIG. 13 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere image is a three-dimensional solid sphere. FIG. 6A is a three-dimensional perspective view of FIG. 5, and FIG. 6B is a diagram showing a state in which an image of a predetermined area is displayed on a display of a communication terminal. 13 is a diagram showing the relationship between predetermined area information and an image of a predetermined area T. FIG. FIG. 2 is a hardware configuration diagram of a near-terminal data processing device or a distributed data processing terminal. FIG. 2 is a hardware configuration diagram of a centralized data processing server. FIG. 2 is a software configuration diagram of a real-time data processing terminal and a near-terminal data processing device in the image acquisition terminal. FIG. 1 is a functional block diagram of a communication system. FIG. 1 is a functional block diagram of a communication system. 1A is a conceptual diagram of an imaging element information management table, and FIG. 1B is a conceptual diagram of a cycle value management table. 1A is a conceptual diagram of an image acquisition program management table, FIG. 1B is a conceptual diagram of a synthesis processing program management table, FIG. 1C is a conceptual diagram of a distortion correction program management table, and FIG. FIG. 13 is a conceptual diagram of an object information management table. FIG. 13 is a conceptual diagram of an object displacement management table. 1A is a conceptual diagram of an object presence management table, and FIG. 1B is a conceptual diagram of a display area management table. 1A is a conceptual diagram of an object management table, FIG. 1B is a conceptual diagram of an object position management table, and FIG. 1C is a conceptual diagram of a count management table. FIG. 2 is a conceptual diagram of a session management table. FIG. 2 is a conceptual diagram of a terminal ID. FIG. 4 is a conceptual diagram of an authentication server management table. FIG. 4 is a conceptual diagram of an authentication management table. FIG. 11 is a sequence diagram showing an authentication process. FIG. 11 is a sequence diagram showing an authentication process. FIG. 13 is a diagram showing an example of a screen of a distributed data processing terminal. FIG. 13 is a diagram showing an example of a screen of a distributed data processing terminal. FIG. 11 is a sequence diagram showing a process of setting a detection area. FIG. 11 is a sequence diagram showing a process of setting a detection area. FIG. 13 is a diagram showing a display area setting screen. FIG. 13 is a diagram showing the positions of detection areas in an actual store. FIG. 13 is a diagram showing a detection range setting screen. FIG. 33(a) is a diagram showing a predetermined area image Q11 before the change in the detectable range P, FIG. 33(b) is a diagram showing a predetermined area image Q111 after the change in the detectable range P, and FIG. 33(c) is a diagram showing an example of a warning display for a predetermined area image Q112 after the change in the detectable range P. FIG. 11 is a sequence diagram showing processing of a request to start image recognition. FIG. 11 is a sequence diagram showing preparation processing for real-time processing of the real-time data processing terminal. FIG. 11 is a sequence diagram showing a program acquisition process. FIG. 11 is a sequence diagram showing an image recognition process. 11 is a flowchart showing object detection processing in real-time processing. 11 is a flowchart showing a process of generating an event in real-time processing. 11 is a flowchart showing a process of generating an event in real-time processing. FIG. 13 is a sequence diagram showing the process of displaying the entry/exit count numbers and storing movement information. 13 is a flowchart showing a change display process in the distributed data processing terminal. FIG. 11 is a sequence diagram showing a matching process. 13 is a flowchart showing a collation process. FIG. 13 is a diagram showing a matching result display screen. FIG. 11 is a sequence diagram showing a process for displaying movement information. 13 is a flowchart showing a process for counting the number of people entering and exiting a building entrance, taking into account the moving direction of an object. FIG. 13 is a diagram showing a movement result display screen.

[Overall Overview]
Hereinafter, an overall outline of one embodiment of the present invention will be described with reference to the drawings. This embodiment discloses a communication system that realizes edge computing. That is, this embodiment discloses an invention that realizes a service provided by a service providing server 2 at an image acquisition terminal 2 through cooperation between processing at a service providing server 8 and processing at an image acquisition terminal 2.

[Overall configuration of the embodiment]
Fig. 1 is a schematic diagram of a communication system according to this embodiment. As shown in Fig. 1, the communication system according to this embodiment is constructed by a real-time data processing terminal 3, a near-terminal data processing device 5, a distributed data processing terminal 6, a centralized data processing server 7, a service providing server 8, and authentication servers 9a, 9b, and 9c. The real-time data processing terminal 3, the near-terminal data processing device 5, and the distributed data processing terminal 6 construct a distributed processing system 100.

The near-terminal data processing device 5 is communicatively connected to the distributed data processing terminal 6 via the intranet 200. The distributed data processing terminal 6 is communicatively connected to the centralized data processing server 7, the service providing server 8, and the authentication servers 9a, 9b, and 9c via the Internet 600. Note that the authentication server 9 is a general term for the authentication servers 9a, 9b, and 9c.

Of these, the real-time data processing terminal 3 is a terminal that performs real-time processing to obtain captured image data. An imaging unit 40 equipped with an image sensor that captures an image of a subject, such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor, is detachably connected to the real-time data processing terminal 3. As a result, the real-time data processing terminal 3 digitizes the captured image data (hereinafter referred to as "captured image data") input from the imaging unit 40, and detects an image of an object (here, a face) in real time (for example, every 1/60 seconds). The real-time data processing terminal 3 transmits data of a partial image (hereinafter referred to as "partial image data"), which is the area of the object part of the captured image, to the near-terminal data processing device 5.

The near-terminal data processing device 5 is placed closest to the real-time data processing terminal 3 and is closely connected one-to-one with the hardware, for example, via a data bus or USB (Universal Serial Bus). The near-terminal data processing device 5 encodes the partial image data received from the real-time data processing terminal 3 into a general-purpose format such as JPEG (Joint Photographic Experts Group), and then transmits it to the distributed data processing terminal 6 via the intranet 200 as data to be matched when face image matching is performed. Note that the real-time data processing terminal 3 and the near-terminal data processing device 5 are usually connected together and integrated, and here the real-time data processing terminal 3 and the near-terminal data processing device 5 constitute the image acquisition terminal 2.

The distributed data processing terminal 6 is a computer that is used by a user at a location relatively close to the near-terminal data processing device 5 and is capable of accepting various operations from the user. The distributed data processing terminal 6 registers and stores matching data in advance when face image matching is performed. The distributed data processing terminal 6 can request the centralized data processing server 7 to match the matching data with the data to be matched via the Internet 600. In this case, the distributed data processing terminal 6 also transmits the data to be matched received from the near-terminal data processing device 5 and the matching data registered in advance. Furthermore, the distributed data processing terminal 6 can receive matching result information indicating the matching result as a response from the centralized data processing server 7. The distributed data processing terminal 6 can also display the matching result via a graphic interface.

The centralized data processing server 7 is installed in a location relatively far from the near-terminal data processing device 5, and can communicate with the distributed data processing terminals 6 via a communication network such as the Internet 600. When the centralized data processing server 7 receives matching request information indicating a matching request, as well as matching data and matched data, it performs a comparison between the matching data and the matched data and determines the degree of similarity. The centralized data processing server 7 then transmits matching result information indicating the matching result, including the degree of similarity, to the distributed data processing terminals 6.

The service providing server 8 is a server that provides various services to the image acquisition terminal 2.

The authentication server 9a authenticates the ID to determine whether the image acquisition terminal 2 has the proper authority to receive services from the service providing system 8. The same is true for the authentication servers 9b and 9c.

[Hardware configuration]
Next, the hardware configuration of the communication system of this embodiment will be described with reference to FIG. 2 to FIG.

<Hardware configuration of real-time data processing terminal>
2 is a diagram showing the hardware configuration of the real-time data processing terminal 3. The real-time data processing terminal 3 includes a CPU 301, a ROM 302, a RAM 303, an EEPROM 304, a CMOS sensor 305, an acceleration/direction sensor 306, a media I/F 308, and a GPS receiving unit 309.

Of these, the CPU 301 controls the overall operation of the real-time data processing terminal 3. The ROM 302 stores programs used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301. The EEPROM 304 reads or writes various data such as programs for the real-time data processing terminal under the control of the CPU 301. The CMOS sensor 305 captures an image of a subject (mainly the blind spot of the photographing unit 40) under the control of the CPU 301 to obtain photographed image data. The acceleration/direction sensor 306 is an electronic magnetic compass that detects geomagnetism, a gyrocompass, an acceleration sensor, or other such sensors. The media I/F 308 controls the reading or writing (storage) of data from or to a recording medium 307 such as a flash memory. The GPS receiver 309 receives GPS signals from GPS satellites.

The real-time data processing terminal 3 also includes an imaging unit I/F 313, a microphone 314, a speaker 315, an audio input/output I/F 316, a display 317, an external device connection I/F 318, and a touch panel 321.

Of these, the imaging unit I/F 313 is a circuit that controls the operation of the imaging unit 40 when an external imaging unit 40 is connected. The microphone 314 is a type of built-in sound collection means that inputs sound. The sound input/output I/F 316 is a circuit that processes the input and output of sound signals between the microphone 314 and the speaker 315 under the control of the CPU 301. The display 317 is a type of display means such as liquid crystal or organic EL that displays images of subjects and various icons. The external device connection I/F 318 is an interface for connecting various external devices. The touch panel 321 is a type of input means that allows the user to operate the real-time data processing terminal 3 by pressing the display 317.

The real-time data processing terminal 3 also includes a bus line 310. The bus line 310 is an address bus, a data bus, etc., for electrically connecting each component such as the CPU 301 shown in FIG. 2.

<Hardware configuration of the imaging unit>
Fig. 3 is a hardware configuration diagram of an imaging unit. In particular, Fig. 3(a) is a hardware configuration diagram of a monocular imaging unit 40a among the imaging units 40. Fig. 3(b) is a hardware configuration diagram of a compound-eye imaging unit 40b among the imaging units 40. Note that imaging unit 40 is a collective term for multiple types of imaging units (imaging units 40a, 40b, etc.) having different numbers of imaging elements.

As shown in FIG. 3(a), the imaging unit 40a has an imaging element 401a such as a CMOS or CCD, a lens 402a, and a connection I/F 408a for electrically connecting to the imaging unit I/F 313 of the real-time data processing terminal 3. When the imaging unit 40a is connected to the imaging unit I/F 313 of the real-time data processing terminal 3, the imaging element 401a captures an image based on an imaging control signal sent from the imaging unit I/F 313 via the connection I/F 408a, and transmits captured image data to the imaging unit I/F 313 via the connection I/F 408a.

As shown in FIG. 3B, the imaging unit 40b has imaging elements 401b1 and 401b2 such as CMOS or CCD, lenses 402b1 and 402b2, and a connection I/F 408b for electrically connecting to the imaging unit I/F 313 of the real-time data processing terminal 3. The lenses 402b1 and 402b2 are, for example, fisheye lenses. When the imaging unit 40b is connected to the imaging unit I/F 313 of the real-time data processing terminal 3, the imaging elements 401b1 and 401b2 capture images based on an imaging control signal sent from the imaging unit I/F 313 via the connection I/F 408b, and transmit multiple captured image data to the imaging unit I/F 313 via the connection I/F 408b.

The imaging unit 40a shown in FIG. 3(a) can obtain a general planar image, while the imaging unit 40b shown in FIG. 3(b) can obtain a spherical image. Here, an outline of the process until an equirectangular projection image EC and a spherical image CE are created from an image captured by the imaging unit 40b will be described with reference to FIG. 4 to FIG. 8. Note that FIG. 4(a) is a diagram showing a hemispherical image (front side) captured by the imaging unit 40b, FIG. 4(b) is a diagram showing a hemispherical image (rear side) captured by the imaging unit 40b, and FIG. 4(c) is a diagram showing an image expressed by equirectangular projection (hereinafter referred to as an "equirectangular projection image"). FIG. 5(a) is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and FIG. 5(b) is a diagram showing a spherical image.

As shown in FIG. 4(a), the image obtained by the image sensor 401b1 becomes a hemispherical image (front side) curved by the lens 402b1. Also, as shown in FIG. 4(b), the image obtained by the image sensor 403b becomes a hemispherical image (rear side) curved by the lens 402b2. Then, the hemispherical image (front side) and the hemispherical image (rear side) flipped 180 degrees are combined to create an equirectangular projection image EC, as shown in FIG. 4(c).

By using OpenGL ES (Open Graphics Library for Embedded Systems), the equirectangular projection image is pasted to cover the sphere as shown in FIG. 5A, and a celestial sphere image CE as shown in FIG. 5B is created. In this way, the celestial sphere image CE is represented as an image in which the equirectangular projection image EC faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the celestial sphere image CE may be a still image or a video.

As described above, the spherical image CE is an image pasted to cover the spherical surface, which gives a sense of incongruity to humans when viewed. Therefore, by displaying a predetermined region of the spherical image CE (hereinafter referred to as a "predetermined region image") as a planar image with little curvature, it is possible to display the image in a way that does not give a sense of incongruity to humans. This will be described with reference to Figs. 6 and 7.

FIG. 6 is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional image is a three-dimensional sphere. The virtual camera IC corresponds to the position of the viewpoint of the user who views the omnidirectional image CE displayed as a three-dimensional sphere. FIG. 7(a) is a three-dimensional perspective view of FIG. 6, and FIG. 7(b) is a diagram showing the predetermined area image when displayed on a display. In FIG. 7(a), the omnidirectional image CE shown in FIG. 5 is represented by a three-dimensional sphere CS. If the omnidirectional image CE generated in this way is a sphere CS, then the virtual camera IC is located inside the omnidirectional image CE as shown in FIG. 6. The predetermined area T in the omnidirectional image CE is the shooting area of the virtual camera IC, and is specified by the predetermined area information indicating the shooting direction and the angle of view of the virtual camera IC in the three-dimensional virtual space including the omnidirectional image CE.

The predetermined area image Q shown in FIG. 7(a) is then displayed on a predetermined display as an image of the shooting area of the virtual camera IC, as shown in FIG. 7(b). The image shown in FIG. 7(b) is a predetermined area image represented by the initial (default) predetermined area information. The following explanation will be given using the shooting direction (ea, aa) and angle of view (α) of the virtual camera IC.

Using FIG. 8, the relationship between the specific area information and the image of the specific area T will be described. FIG. 8 is a diagram showing the relationship between the specific area information and the image of the specific area T. As shown in FIG. 8, "ea" indicates the elevation angle, "aa" indicates the azimuth angle, and "α" indicates the angle of view. That is, the posture of the virtual camera IC is changed so that the gaze point of the virtual camera IC indicated by the shooting direction (ea, aa) becomes the center point CP of the specific area T, which is the shooting area of the virtual camera IC. The specific area image Q is an image of the specific area T in the omnidirectional image CE. f is the distance from the virtual camera IC to the center point CP. L is the distance between any vertex of the specific area T and the center point CP (2L is the diagonal). In FIG. 8, the trigonometric function generally shown in the following (Equation 1) holds. The center point CP can be identified by the shooting direction (ea, aa) and distance f, so the specified area information can be expressed by the center point CP and the angle of view α.

<Hardware configuration of near-terminal data processing device and distributed data processing terminal>
9 is a hardware configuration diagram of the near-terminal data processing device or the distributed data processing terminal. Here, the near-terminal data processing device 5 and the distributed data processing terminal 6 are shown to have the same hardware configuration, so the near-terminal data processing device 5 will be described and the description of the distributed data processing terminal 6 will be omitted.

As shown in FIG. 9, the near-terminal data processing device 5 includes a CPU 501, a ROM 502, a RAM 503, an EEPROM 504, a CMOS sensor 505, an acceleration/direction sensor 506, a media I/F 508, and a GPS receiver 509.

Of these, the CPU 501 controls the overall operation of the near-terminal data processing device 5. The ROM 502 stores programs used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The EEPROM 504 reads or writes various data such as programs for the near-terminal data processing device under the control of the CPU 501. The CMOS sensor 505 captures an image of a subject (mainly a self-portrait of the user operating the near-terminal data processing device 5) under the control of the CPU 501, and obtains captured image data. The acceleration/direction sensor 506 is an electronic magnetic compass that detects geomagnetism, a gyrocompass, an acceleration sensor, and other sensors. The media I/F 508 controls the reading or writing (storage) of data from the recording media 507 such as a flash memory. The GPS receiver 509 receives GPS signals from GPS satellites.

The near-terminal data processing device 5 also includes a long-distance communication circuit 511, an antenna 511a of the long-distance communication circuit 511, a camera 512, an image sensor I/F 513, a microphone 514, a speaker 515, an audio input/output I/F 516, a display 517, an external device connection I/F 518, a short-distance communication circuit 519, an antenna 519a of the short-distance communication circuit 519, and a touch panel 521.

Among these, the long-distance communication circuit 511 is a circuit that communicates with other devices via the intranet 200. The camera 512 is a type of built-in imaging means that captures an image of a subject under the control of the CPU 501 to obtain captured image data. The image sensor I/F 513 is a circuit that controls the operation of the camera 512. The microphone 514 is a type of built-in sound collection means that inputs sound. The sound input/output I/F 516 is a circuit that processes the input/output of sound signals between the microphone 514 and the speaker 515 under the control of the CPU 501. The display 517 is a type of display means such as liquid crystal or organic EL that displays an image of a subject, various icons, etc. The external device connection I/F 518 is an interface for connecting various external devices. The short-distance communication circuit 519 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The touch panel 521 is a type of input means that allows a user to operate the near-terminal data processing device 5 by pressing the display 517.

The near-terminal data processing device 5 also includes a bus line 510. The bus line 510 is an address bus, a data bus, etc., for electrically connecting each component such as the CPU 501 shown in FIG. 9.

<Hardware configuration of centralized data processing server, service providing server, and authentication server>
10 is a hardware configuration diagram of the centralized data processing server, the service providing server, or the authentication server. Here, the centralized data processing server 7, the service providing server 8, and the authentication server 9 are shown to have the same hardware configuration, so only the centralized data processing server 7 will be described and descriptions of the service providing server 8 and the authentication server 9 will be omitted.

Figure 10 is a hardware configuration diagram of the centralized data processing server. The centralized data processing server 7 is constructed by a computer, and as shown in Figure 10, is equipped with a CPU 701, ROM 702, RAM 703, HD 704, HDD (Hard Disk Drive) 705, recording media 706, media I/F 707, display 708, network I/F 709, keyboard 711, mouse 712, DVD-RW drive 714, and bus line 710. Note that since the centralized data processing server 7 functions as a server, it does not need input devices such as the keyboard 711 and mouse 712, or output devices such as the display 708.

Of these, the CPU 701 controls the operation of the entire centralized data processing server 7. The ROM 702 stores programs used to drive the CPU 701. The RAM 703 is used as a work area for the CPU 701. The HD 704 stores various data such as programs. The HDD 705 controls the reading or writing of various data from the HD 704 according to the control of the CPU 701. The media I/F 707 controls the reading or writing (storage) of data from a recording medium 706 such as a flash memory. The display 708 displays various information such as a cursor, menu, window, character, or image. The network I/F 709 is an interface for data communication using the Internet 600. The keyboard 711 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The mouse 712 is a type of input means for selecting and executing various instructions, selecting a processing target, moving the cursor, etc. The DVD-RW drive 714 controls the reading of various data from a DVD-RW (Digital Versatile Disc-ReWritable) 713, which is an example of a removable recording medium.

The centralized data processing server 7 also includes a bus line 710. The bus line 710 is an address bus, a data bus, etc., for electrically connecting each component such as the CPU 701 shown in FIG. 10.

[Software configuration]
FIG. 11 is a software configuration diagram of the real-time data processing terminal and the near-terminal data processing device in the image acquisition terminal.

As shown in FIG. 11, the real-time data processing terminal 3 has an OS 300 and an image recognition application (hereinafter, "application" will be referred to as "app") AP1. The image recognition app AP1 runs on the RAM 303 of the real-time data processing terminal 3. Of these, the OS 300 is basic software that provides basic functions and manages the entire real-time data processing terminal 3. The image recognition app AP1 is an app for recognizing the faces of people, animals, etc. from captured images.

The near-terminal data processing device 5 also has an OS 500 and a communication application AP2. The communication application AP2 runs on the RAM 503 of the near-terminal data processing device 5. Of these, the OS 500 is basic software that provides basic functions and manages the entire near-terminal data processing device 5. The communication application AP2 is an application for communicating with other terminals (devices) such as the distributed data processing terminal 6.

In this way, in the image acquisition terminal 2, the real-time data processing terminal 3 performs image recognition, while the near-terminal data processing device 5 communicates with the distributed data processing terminal 6 via the intranet 200, thereby enabling distributed processing to be performed. The distributed processing system 100 including the image acquisition terminal 2, the intranet 200, and the distributed data processing terminal 6 is an example of a processing system.

In addition, the real-time data processing terminal 3 and the near-terminal data processing device 5 are equipped with not only an OS, but also a driver, a software development kit (SDK), or an application programming interface (API), which may be different from each other.

Next, the functional configuration and processing of one embodiment of the present invention will be explained using the drawings.

[Functional configuration of the embodiment]
First, the functional configurations of the terminals, devices, and servers constituting the communication system according to the present embodiment will be described with reference to Fig. 12 to Fig. 23. Fig. 12 is a functional block diagram of the communication system according to the embodiment. Fig. 12 particularly shows the functional blocks of the image acquisition terminal 2.

<Functional configuration of real-time data processing terminal>
12, the real-time data processing terminal 3 has a calculation unit 31, a judgment unit 33, an image processing unit 34, an object detection unit 35, an event generation unit 36, a display control unit 37, a connection unit 38, a storage/readout unit 39, and a communication unit 48. Each of these units is a function or means realized by the operation of any of the components shown in FIG. 2 by an instruction from the CPU 301 in accordance with the program loaded from the EEPROM 304 onto the RAM 303.

The real-time data processing terminal 3 also has a memory unit 3000 constructed from the ROM 302, RAM 303, and EEPROM 304 shown in FIG. 2. Shape model data, which will be described later, is stored in the memory unit 3000. Furthermore, the memory unit 3000 also contains an image sensor information management DB 3001, a cycle value management DB 3002, an image acquisition program management DB 3003, a synthesis processing program management DB 3004, a distortion correction program management DB 3005, a service program management DB 3006, an object information management DB 3007, and an object displacement management DB 3008.

The imaging element information management DB 3001 is composed of an imaging element information management table, which will be described later. The cycle value management DB 3002 is composed of a cycle value management table, which will be described later. The image acquisition program management DB 3003 is composed of an image acquisition program management table, which will be described later. The synthesis processing program management DB 3004 is composed of a synthesis processing program management table, which will be described later. The distortion correction program management DB 3005 is composed of a distortion correction program management table, which will be described later. The service program management DB 3006 is composed of a service program management table, which will be described later.

Furthermore, the object information management DB 3007 is configured from an object information management table described below. The object displacement management DB 3008 is configured from an object displacement management table described below.

(Imaging element information management table)
14A is a conceptual diagram showing an image sensor information management table. In this image sensor information management table, the model number of the image sensor unit 40, the number of image sensors included in the image sensor unit 40, and the type of lens used in the image sensor are managed in association with each other. The model number is an example of type information that indicates the type of image sensor unit according to the number of image sensors and the type of lens used in the image sensor.

(Cycle value management table)
14B is a conceptual diagram showing a cycle value management table, in which the number of image pickup elements included in the image pickup unit 40 and a cycle value (frames per second) of object recognition processing, which will be described later, are associated and managed.

Each table shown in FIG. 15 is a table for managing whether or not each program is installed on the real-time data processing terminal 3.

(Image Acquisition Program Management Table)
15A is a conceptual diagram showing an image acquisition program management table. In this image acquisition program management table, the number of imaging elements in the imaging unit 40 and the image acquisition programs (or their names) are managed in association with each other. For example, when an image acquisition program for the number of imaging elements "1" is installed in the real-time data processing terminal 3, the program "ProgC01 (image acquisition for one system)" is managed in association with each other. When an image acquisition program for the number of imaging elements "2" is installed in the real-time data processing terminal 3, the program "ProgC02 (image acquisition for two systems)" is managed in association with each other.

(Mixing Processing Program Management Table)
15B is a conceptual diagram showing a compositing process program management table, which manages the number of image pickup elements included in the image pickup unit 40 and the names of compositing processes in association with each other.

(Distortion correction program management table)
15C is a conceptual diagram showing a distortion correction program management table. In this distortion correction program management table, the type of lens of the imaging unit 40 and the distortion correction program (or the name thereof) are managed in association with each other. For example, when a distortion correction program for the lens type "wide-angle" is installed in the real-time data processing terminal 3, the program "ProgW01 (wide-angle distortion correction)" is managed in association with each other. When a distortion correction program for the lens type "fisheye" is installed in the real-time data processing terminal 3, the program "ProgW02 (fisheye distortion correction)" is managed in association with each other.

(Service program management table)
15D is a conceptual diagram showing a service program management table. In this service program management table, an authentication server ID for identifying an authentication server and the name of a service program for executing a service provided by the image acquisition terminal 2 are managed in association with each other. For example, when a service program for authentication server ID "a01" is installed in the real-time data processing terminal 3, a program "ProgD01 (object detection)" is managed in association with each other. Also, when a service program for authentication server ID "a02" is installed in the real-time data processing terminal 3, a program "ProgD02 (object counting)" is managed in association with each other.

(Object Information Management Table)
16 is a conceptual diagram showing an object information management table. In this object information management table, a (serial number), a position of a rectangular partial image in a detectable range P described later, a width of the partial image, a height of the partial image, and an end flag (here, "*") are managed in association with each other. The position of the partial image indicates the coordinate position of the upper left corner of the rectangular image.

(Object Displacement Management Table)
17 is a conceptual diagram showing an object displacement management table. In this object displacement management table, the image ID of a partial image for identifying a partial image, the position (coordinates) of this partial image in the collation possible range C, the width of the partial image, the height of the partial image, the collation progress status indicating the progress status of the partial image being collated in the centralized data processing server 7, the object ID for identifying an object (here, a person) shown in the partial image, and the end flag (here, "*") are managed in association with each other. Among these, the position of the partial image in the detectable range P, the width of the partial image, and the height of the partial image are object identification information transferred from the object information management table (see FIG. 16). That is, the information (position, width, height) in the object information management table is partial image identification information for identifying the current partial image, and the information (position, width, height) in the object displacement management table is partial image identification information for identifying the past partial image.

The object ID is an example of object identification information for identifying an object. Examples of object identification information include employee numbers, student ID numbers, driver's license numbers, and the My Number used in the Japanese system.

In addition, the "end flag" in the object information management table (see FIG. 16) is set to indicate that the judgment is completed when it is determined that the current position of an object in any record in the object information management table is within a predetermined distance (threshold) from the past position of the object in each record of the object displacement management table (see FIG. 17). On the other hand, the "end flag" in the object displacement management table is set to the record containing partial image identifying information transferred from the object information management table when this partial image identifying information is present.

(Functional configuration of real-time data processing terminal)
Next, each functional configuration of the real-time data processing terminal 3 will be described in more detail with reference to FIG.

The calculation unit 31 of the real-time data processing terminal 3 is realized by processing of the CPU 301, and calculates the distance between the latest position of an object and the past position of the object.

The determination unit 33 is realized by the processing of the CPU 301, and performs various determinations. For example, the determination unit 33 determines the number of imaging elements based on the model number sent from the imaging unit 40, with reference to the imaging element information management DB 3001.

The image processing unit 34 is realized by the processing of the CPU 301, and performs various image processing by executing each program shown in FIG. 15 (image acquisition program, image synthesis program, distortion correction program, and service program). Specifically, the image processing unit 34 performs first image processing (e.g., image acquisition, image synthesis, distortion correction) on image data by executing a first program (e.g., image acquisition program, image synthesis program, and distortion correction program) that does not require authentication to be acquired. The image processing unit 34 also performs second image processing (e.g., object detection processing, object counting processing) on image data by executing a second program (e.g., service program) that requires authentication to be acquired.

The object detection unit 35 is realized by processing of the CPU 301, and detects feature points that are candidates for a predetermined specific object such as a face in the captured image data acquired by the image processing unit 34, and detects the position of the object in the captured image by referring to shape model data that indicates a shape model of the predetermined specific object such as a face.

The event generation unit 36 is realized by processing of the CPU 301, and generates detection information (event information) that indicates that the object position has been detected by the object detection unit 35.

The display control unit 37 is realized by the processing of the CPU 301, and causes various screens to be displayed on the display 317, which is an example of a display means.

The connection unit 38 is realized by the processing of the imaging unit I/F 313 and the CPU 301, and is an interface for mechanically and electrically connecting the imaging unit 40 to the real-time data processing terminal 3.

The storage/reading unit 39 is realized by the processing of the CPU 301, and stores various data (or information) in the storage unit 3000 and reads various data (or information) from the storage unit 3000.

The communication unit 48 is realized by the processing of the external device connection I/F 318 and the CPU 301, and transmits and receives various data (or information) to and from a communication unit 58 (described later) of the near-terminal data processing device 5 via one-to-one communication. Note that this communication may be wireless as well as wired.

<Functional configuration of near-terminal data processing device>
12, the near-terminal data processing device 5 has a transmitting/receiving unit 51, a data detecting unit 56, a display control unit 57, a communication unit 58, and a storing/reading unit 59. Each of these units is a function or means realized by the operation of any of the components shown in FIG. 9 by an instruction from the CPU 501 in accordance with a program loaded from the EEPROM 504 onto the RAM 503 in the near-terminal data processing device 5.

The near-terminal data processing device 5 also has a memory unit 5000 constructed from a ROM 502, a RAM 503, and an EEPROM 504 as shown in FIG. 9.

(Functional configuration of near-terminal data processing device)
Next, the functional configuration of the near-terminal data processing device 5 will be described in more detail with reference to FIG.

The transmitter/receiver 51 of the near-terminal data processing device 5 is realized by the processing of the long-distance communication circuit 511, the antenna 511a, and the CPU 501, and transmits and receives various data (or information) with the distributed data processing terminal 6 via a communication network (here, the intranet 200).

The data detection unit 56 is realized by the processing of the CPU 501, and detects whether an event has occurred that requires the reception of data from the real-time data processing terminal 3, and whether the reception of the data has been completed.

The display control unit 57 is realized by processing of the CPU 501, and causes various screens to be displayed on the display 517, which is an example of a display means.

The communication unit 58 is realized by the processing of the external device connection I/F 518 and the CPU 501, and transmits and receives various data (or information) to and from the communication unit 48 of the real-time data processing terminal 3 via one-to-one communication. This communication may be wireless or wired.

The storage/read unit 59 is realized by the processing of the CPU 501, and stores various data (or information) in the storage unit 5000 and reads various data (or information) from the storage unit 5000.

<Functional configuration of distributed data processing terminal>
13, the distributed data processing terminal 6 has a transmitting/receiving unit 61, a receiving unit 62, a determining unit 63, a calculating unit 65, a display control unit 67, and a storing/reading unit 69. Each of these units is a function or means realized by the operation of any of the components shown in FIG. 9 in the distributed data processing terminal 6 by an instruction from the CPU 501 in accordance with a program loaded from the EEPROM 504 onto the RAM 503.

The distributed data processing terminal 6 also has a memory unit 6000 constructed from the ROM 502, RAM 503, and EEPROM 504 shown in FIG. 9. The memory unit 6000 contains an object presence management DB 6001 and a display area management DB 6002.

The object presence management DB 6001 is composed of an object presence management table, which will be described later. The display area management DB 6002 is composed of a display area management table.

(Object Presence Management Table)
18A is a conceptual diagram showing an object presence management table provided in the object presence management DB 6001. In this object presence management table, object IDs for identifying objects such as people are managed. This makes it possible to manage the presence of an object identified by an object ID in any area (areas a1 to a4 shown in FIG. 31 described later) in the store.

(Display Area Management Table)
Fig. 18B is a conceptual diagram showing a display area management table provided in the display area management DB 6002. This display area management table manages areas within the store that are displayed in each of the detection area display fields d11 to d14 shown in Fig. 30 described later.

(Functional configuration of distributed data processing terminal)
The transmitting/receiving unit 61 of the distributed data processing terminal 6 is realized by the processing of the long-distance communication circuit 511, antenna 511a, and CPU 501 in the distributed data processing terminal 6, and transmits and receives various data (or information) to and from the centralized data processing server 7 via a communication network (here, the Internet 600). For example, the transmitting/receiving unit 61 transmits a request for matching between matching data and matched data to the centralized data processing server 7, and processes the matching result sent from the centralized data processing server 7.

The reception unit 62 is realized by the processing of the touch panel 521 and the CPU 501 in the distributed data processing terminal 6, and receives various operations from the user.

The judgment unit 63 is realized by the processing of the CPU 501 in the distributed data processing terminal 6, and performs various judgments.

The calculation unit 65 is realized by the processing of the CPU 501 in the distributed data processing terminal 6, and performs various calculations.

The display control unit 67 is realized by processing of the CPU 501 in the distributed data processing terminal 6, and causes various screens to be displayed on the display 517, which is an example of a display means.

The storage/reading unit 69 is realized by the processing of the CPU 501 in the distributed data processing terminal 6, and stores various data (or information) in the storage unit 6000 and reads various data (or information) from the storage unit 6000. For example, in response to a registration request accepted by the acceptance unit 62, the storage/reading unit 69 stores and registers matching data (here, facial image data) in the storage unit 6000.

<Functional configuration of the centralized data processing server>
13, the centralized data processing server 7 has a transmission/reception unit 71, a reception unit 72, a judgment unit 73, a feature generation unit 74, a collation unit 75, a counting unit 76, and a storage/readout unit 79. Each of these units is a function or means realized by the operation of any of the components shown in Fig. 10 by an instruction from the CPU 701 in accordance with a program loaded from the HD 704 onto the RAM 703 in the distributed data processing terminal 6.

The centralized data processing server 7 also has a storage unit 7000 constructed by the ROM 702, RAM 703, and HD 704 shown in FIG. 10. The storage unit 7000 stores feature data on the matching side, which will be described later. This storage unit 7000 stores matching data. Furthermore, various DBs 7001, 7002, and 7003 are constructed in the storage unit 7000. The matching data management DB 7001 is constructed by a matching data management table, which will be described later. Note that the matching data may be stored in a data management server other than the centralized data processing server 7. Furthermore, the storage unit 7000 also has an object position management DB 7002 constructed. The object position management DB 7002 is constructed by an object position management table, which will be described later. Furthermore, the storage unit 7000 also has a counting management DB 7003 constructed. The counting management DB 7003 is constructed by a counting management table, which will be described later.

(Verification data management table)
19A is a conceptual diagram showing a matching data management table provided in the matching data management DB 7001. In this matching data management table, an image file indicating the file name of the matching data and an object ID for identifying an object, such as a person, shown in the matching image related to this image file are managed in association with each other.

(Object Position Management Table)
FIG. 19(b) is a conceptual diagram showing an object location management table provided in the object location management DB 7002. In this object location management table, the reception date and time, area number, and object ID are managed in association with each other. Of these, the reception date and time indicates the date and time when the centralized data processing server 7 received a movement information storage request from the distributed data processing terminal 6 (see S306). The area number is information indicating the area within the store where the object as a person was present. For example, area number "1" indicates that the person was present (located) in area a1 within the store shown in FIG. 31. The object ID is the same as the object ID in the matching data management table.

(Counting Management Table)
19C is a conceptual diagram showing a counting management table provided in the counting management DB 7003. In this counting management table, the number of people or other objects that have moved to a movement area (left, up, right, down) is associated and managed for each time period (e.g., one hour) in the store.

(Functional configuration of the centralized data processing server)
The transmitting/receiving unit 71 of the centralized data processing server 7 is realized by the processing of the network I/F 709 and the CPU 701, and transmits and receives various data (or information) to and from the distributed data processing terminals 6 via a communication network (here, the Internet 600). The transmitting/receiving unit 71 receives a request for matching between matching data and matched data from the distributed data processing terminals 6, and transmits matching result information indicating the matching result to the distributed data processing terminals 6.

The reception unit 72 is realized by the processing of the keyboard 711, mouse 712, and CPU 701, and receives various operations from the user.

The judgment unit 73 is realized by the processing of the CPU 501 in the centralized data processing server 7, and performs various judgments.

The feature generation unit 74 is realized by processing of the CPU 701, and generates feature parameters from the matched data (partial image data) and matching data received by the transmission/reception unit 71.

The matching unit 75 is implemented by the processing of the CPU 701, and uses the features generated by the feature generation unit 74 to match the features of the matching data and the features of the data to be matched, and calculates a score (points) indicating the degree of similarity.

The counting unit 76 is implemented by the processing of the CPU 701, and counts the number of records for each combination of direction and object ID for each unit time (each hour in this embodiment) in the movement information stored in the object position management DB 7002.

The storage/read unit 79 is realized by the processing of the CPU 701, and stores various data (or information) in the storage unit 7000 and reads various data (or information) from the storage unit 7000.

<Functional configuration of the service providing server>
Next, each functional configuration of the service providing server 8 will be described in detail with reference to Fig. 13 and Fig. 20 to Fig. 22. As shown in Fig. 13, the service providing server 8 has a transmission/reception unit 81, a determination unit 82, an extraction unit 87, and a storage/readout unit 89. Each of these units is a function or means realized by any of the components shown in Fig. 10 operating in response to an instruction from the CPU 701 in accordance with the program for the service providing server 8 loaded from the HD 704 onto the RAM 703.

The service providing server 8 also has a memory unit 8000 constructed by the RAM 703 and HD 704 shown in FIG. 10. This memory unit 8000 stores various data sent from the distributed data processing terminal 6 or the authentication server 9. The memory unit 8000 also stores all of the programs shown in FIG. 15, and the service providing server 8 can transmit the requested program in response to a request from the real-time data processing terminal 3.

Furthermore, the memory unit 8000 has a session management DB 8001 and an authentication server management DB 8002. Of these, the session management DB 8001 is configured with a session management table, which will be described later. The authentication server management DB 8002 is configured with an authentication server management table, which will be described later. Each table will be described in detail below.

(Session Management Table)
20 is a conceptual diagram showing a session management table provided in the session management DB 8001. In this session management table, a session ID for identifying a communication session to be established with the distributed data processing terminal 6 in order to provide a service to the distributed data processing terminal 6, a terminal ID for identifying a user of the distributed data processing terminal 6, and an IP address of the distributed data processing terminal 6 of the user identified by the terminal ID are stored and managed in association with each other.

Figures 21 (a), (b), and (c) show an email address as an example of a terminal ID (identification), and each is composed of an authentication subject portion and a non-authentication subject portion. The authentication subject portion is a user ID that is used when authenticating by the authentication server 9. The non-authentication subject portion is a portion that is not used when authenticating by the authentication server 9.

In the first pattern shown in FIG. 21(a), the part to be authenticated is composed of the account name "asai", the host name "myhost", and the first part of the domain name "ricoo.com". On the other hand, the part not to be authenticated is composed of the last part of the domain name "theta1". In this case, the extraction unit 87 distinguishes between the part to be authenticated and the part not to be authenticated by the "/".

Figure 21(b) also shows the first pattern, but the part not subject to authentication is different from that shown in Figure 21(a). That is, the authentication server 9 authenticates the terminal ID shown in Figure 21(a) and the terminal ID shown in Figure 21(b) as the same ID because the part subject to authentication is the same.

The terminal ID may be a second pattern shown in FIG. 21(c). In this second pattern, the part to be authenticated is composed of the first part of the account name, "asai". On the other hand, the part not to be authenticated is composed of the last part of the account name, "theta2", the host name, "myhost", and the domain name, "ricoo.com". In this case, the extraction unit 87 distinguishes between the part to be authenticated and the part not to be authenticated by using "+".

(Authentication Server Management Table)
22 is a conceptual diagram showing an authentication server management table, in which a uniform resource locator (URL) for accessing each authentication server 9 is stored and managed in association with each authentication server ID for identifying each authentication server 9.

(Functional configuration of the service providing server)
Next, each functional configuration of the service providing server 8 will be described in detail with reference to FIG.

The transmission/reception unit 81 of the service providing server 8 is realized mainly by commands from the CPU 701 shown in FIG. 10 and the network I/F 709, and transmits and receives various data (or information) with the distributed data processing terminal 6 or the authentication server 9 via the Internet 600.

The judgment unit 82 is mainly realized by instructions from the CPU 701 shown in FIG. 10, and judges, for example, whether a communication session for providing a service to the distributed data processing terminal 6 has already been established.

The extraction unit 87 is mainly realized by commands from the CPU 701 shown in FIG. 10, and performs processing to extract the user ID (part to be authenticated) as shown in FIG. 21 from the terminal ID.

The storage/read unit 89 is realized mainly by instructions from the CPU 701 shown in FIG. 10 and the HDD 705, and stores various data in the storage unit 8000 and reads various data from the storage unit 8000.

<Functional configuration of authentication server>
Next, the functional configuration of the authentication server 9 will be described in detail with reference to Fig. 13 and Fig. 23. The authentication server 9 has a transmission/reception unit 91, an authentication unit 92, and a storage/readout unit 99. Each of these units is a function or means realized by any of the components shown in Fig. 10 operating in response to an instruction from the CPU 701 in accordance with the program for the authentication server 9 loaded from the HD 704 onto the RAM 703.

The authentication server 9 also has a memory unit 9000 constructed by the RAM 703 and HD 704 shown in FIG. 10. This memory unit 9000 stores various data sent from the distributed data processing terminal 6 or the service providing server 8.

Furthermore, an authentication management DB 9001 is constructed in the memory unit 9000. The authentication management DB 9001 is composed of an authentication management table, which will be described later. This table will be described in detail below.

(Authentication Management Table)
Figure 23(a) is a conceptual diagram showing an authentication management table held by authentication server 9a, Figure 23(b) is a conceptual diagram showing an authentication management table held by authentication server 9b, and Figure 23(c) is a conceptual diagram showing an authentication management table held by authentication server 9c.

In each authentication management table, the user ID (the part to be authenticated) of the terminal ID and the password are stored and managed in association with each other.

(Functional configuration of the authentication server)
Next, each functional configuration of the authentication server 9 will be described in detail with reference to FIG.

The transmission/reception unit 91 of the authentication server 9 is realized mainly by commands from the CPU 701 shown in FIG. 10 and the network I/F 709, and transmits and receives various data (or information) with the distributed data processing terminal 6 and the service providing server 8 via the Internet 600.

The authentication unit 92 is mainly realized by commands from the CPU 701 shown in FIG. 10, and performs ID authentication by determining whether the image acquisition terminal 2 that sent the authentication request has the legitimate authority to receive the service.

The storage/reading unit 99 is realized mainly by instructions from the CPU 701 shown in FIG. 10 and the HDD 705, and stores various data (or information) in the storage unit 9000 and reads various data (or information) from the storage unit 9000.

[Processing or Operation of the Present Embodiment]
Next, the processing or operation of this embodiment will be described with reference to FIGS.

<Authentication process>
First, the authentication process will be described with reference to Fig. 24 to Fig. 27. Fig. 24 and Fig. 25 are sequence diagrams showing the authentication process. Fig. 26 and Fig. 27 are diagrams showing examples of screens of the distributed data processing terminal.

As shown in FIG. 24, the transmission/reception unit 61 of the distributed data processing terminal 6 transmits a request for an authentication destination selection screen to the service providing server 8 (step S21). This request includes the terminal ID of the distributed data processing terminal 6. At this time, the transmission/reception unit 61 also transmits the IP address of its own terminal. As a result, the transmission/reception unit 81 of the service providing server 8 receives the request for the authentication destination selection screen and the IP address of the distributed data processing terminal 6.

Next, the determination unit 82 of the service providing server 8 determines whether or not the terminal ID received in step S21 is associated with a specific session ID and managed in the session management table (see FIG. 20) (step S22). Next, a case where the terminal ID is not managed will be described.

The transmission/reception unit 81 of the service providing server 8 transmits the data of the authentication destination selection screen to the distributed data processing terminal 6 (step S23). As a result, the transmission/reception unit 61 of the distributed data processing terminal 6 receives the data of the authentication destination selection screen.

Next, the display control unit 67 of the distributed data processing terminal 6 causes the display 517 to display an authentication destination selection screen s1 as shown in FIG. 26 (step S24). FIG. 26 shows an example screen of the distributed data processing terminal 6. This authentication destination selection screen s1 displays an input field b1 for a terminal ID, an input field b2 for a password, and a login button b3 for making a login request (authentication request). Furthermore, the authentication destination selection screen s1 displays authentication server selection buttons a1, a2, and a3 for selecting authentication servers 9a, 9b, and 9c, respectively. For example, the authentication server selection button a1 is a button for when the user receives the object detection processing service. The authentication server selection button a2 is a button for when the user receives the object counting processing service.

When the user inputs his/her terminal ID in input field b1, his/her password in input field b2, presses the desired button among authentication server selection buttons a1, a2, a3, and presses login button b3, the reception unit 62 accepts each input and selection (step S25). Here, a case will be described in which the object detection process is executed by the object detection process program ProgD01 as a result of the authentication server selection button a1 being selected, out of the image recognition process.

The transmitting/receiving unit 61 transmits an authentication request for the ID (here, the terminal ID or user ID) to the service providing server 8 (step S26). This authentication request includes the terminal ID and password accepted in step S25, as well as the selection result of the authentication server 9 and the URL of the distributed data processing terminal 6. This authentication result indicates the authentication server ID for identifying the authentication server 9. As a result, the transmitting/receiving unit 81 of the service providing server 8 receives the ID authentication request.

Next, the storage/reading unit 89 of the service providing server 8 searches the authentication server management table (see FIG. 22) using the authentication server ID received as the selection result in step S26 as a search key to read out the URL of the corresponding authentication server (step S27).

Next, the extraction unit 87 extracts only the user ID (part to be authenticated) from the terminal ID received in step S26 (step S28). The transmission/reception unit 81 then transmits an ID authentication request to the authentication server 9 indicated by the URL read in step S27 (step S29). This ID authentication request includes the user ID (part to be authenticated) extracted in step S28, the password received in step S26, and the URL of the distributed data processing terminal 6 received in step S26. As a result, the transmission/reception unit 71 of the authentication server 9 receives the user authentication request.

Then, the storage/read unit 99 of the authentication server 9 uses the pair of the user ID (part to be authenticated) and password received in step S29 as a search key, and the authentication unit 92 performs authentication using the results of searching the authentication management table (see FIG. 23) for the same pair of the part to be authenticated and the password (step S30). If the same pair is managed, the authentication unit 92 determines that the distributed data processing terminal 6 is a legitimate terminal for receiving services from the service providing server 8, and if the same pair is not managed, the authentication unit 92 determines that the distributed data processing terminal 6 is not a legitimate terminal for receiving services from the service providing server 8.

In step S28, the extraction unit 87 extracts the part to be authenticated from the terminal ID, but this is not limited to the above. For example, the service providing server 8 may not have an extraction unit 87, and in step S29, the transmission/reception unit 81 may transmit only the user ID (part to be authenticated) of the terminal ID in addition to the password and URL.

Next, as shown in FIG. 25, the authentication unit 92 of the authentication server 9 encrypts the token (transmission right) (step S31). Then, the transmission/reception unit 91 transmits the authentication result to the distributed data processing terminal 6 based on the URL of the distributed data processing terminal 6 received in step S29 (step S32). This authentication result indicates whether the distributed data processing terminal 6 is legitimate or not, and includes the token encrypted in step S31. As a result, the transmission/reception unit 61 of the distributed data processing terminal 6 receives the authentication result of the user. Below, we will explain the case where the user has legitimate authority.

The transmission/reception unit 61 of the distributed data processing terminal 6 transmits a session establishment request to the service providing server 8 (step S33). This session establishment request includes the terminal ID and the encrypted token received in step S32. As a result, the transmission/reception unit 81 of the service providing server 8 receives the session establishment request.

Next, in order to confirm that the distributed data processing terminal 6 that sent the session establishment request was determined to be valid in step S30, the transmission/reception unit 81 of the service providing server 8 sends a token authentication request to the authentication server 9 (step S34). This token authentication request includes the encrypted token received in step S33. As a result, the transmission/reception unit 91 of the authentication server 9 receives the token authentication request.

Next, the authentication unit 92 decrypts the encrypted token received in step S34 (step S35). The authentication unit 92 then authenticates the token by comparing the token before encryption in step S31 with the token after decryption in step S35 (step S36). The transmission/reception unit 91 of the authentication server 9 then transmits the authentication result of step S36 to the service providing server 8 (step S37). As a result, the transmission/reception unit 81 of the service providing server 8 receives the authentication result. Below, a case where the token is determined to be valid in step S36 will be described.

Next, the storage/read unit 89 of the service providing server 8 assigns a new session ID in the session management table (see FIG. 20), and manages this session ID in association with the terminal ID and IP address received in step S26 (step S38). The transmission/reception unit 81 then transmits data of the service providing screen to the distributed data processing terminal 6 (step S39). As a result, the transmission/reception unit 61 of the distributed data processing terminal 6 receives the data of the service providing screen.

Next, the display control unit 67 of the distributed data processing terminal 6 displays a service provision screen s2 as shown in FIG. 27 on the display 517 (step S40). FIG. 27 shows an example of a screen of the distributed data processing terminal 6. Here, as an example of the service provided, a remote control service for realizing remote control of the image acquisition terminal 2 from the distributed data processing terminal 6 is described. The service provision screen s2 shown in FIG. 27 displays an IP address input field c1 for identifying the remote control target, a "Start remote control" button c2, and a "Display area management" button c3.

<Detection area setting>
Next, the process of setting the detection area will be described with reference to Fig. 28 to Fig. 32 and Fig. 38(a). Fig. 28 and Fig. 29 are sequence diagrams showing the process of setting the detection area.

First, as shown in FIG. 28, the reception unit 62 of the distributed data processing terminal 6 receives a detection area setting request from the user by pressing the "detection area setting" button c3 shown in FIG. 27 (step S41). Then, the transmission/reception unit 61 transmits the detection area setting request to the near-terminal data processing device 5 (step S42). As a result, the transmission/reception unit 51 of the near-terminal data processing device 5 receives the detection area setting request.

Next, the communication unit 58 of the near-terminal data processing device 5 outputs the detection area setting request received in step S42 to the communication unit 48 of the real-time data processing terminal 3 (step S43). As a result, the communication unit 48 of the real-time data processing terminal 3 inputs the detection area setting request.

Next, the real-time data processing terminal 3 executes object detection processing in display area management as part of real-time processing (step S44).

(Object detection processing in display area management)
Here, the object detection process in the display area management among the real-time processes will be described with reference to Fig. 38(a). Fig. 38(a) is a flowchart showing the object detection process in the display area management among the real-time processes.

First, the determination unit 33 determines whether the number of imaging elements in the connected imaging unit 40 is one (step S201). In this case, the determination unit 33 makes a determination based on the number of imaging elements read out by the processing of the above-mentioned step S72. Then, if the number of imaging elements is one (step S201; YES), the image processing unit 34 sets the cycle value for repeating the process to 1/60 seconds (step S202). In this case, the image processing unit 34 sets the cycle value read out by the processing of the above-mentioned step S73.

Next, the connection unit 38 acquires one line of captured image data from the imaging unit 40 (step S203). This captured image data is digital image data, for example, 4K image data (3840 pixels wide x 2160 pixels high). In this case, the connection unit 38 executes processing using the image acquisition program (ProgC01 (one line)) shown in FIG. 15(a).

On the other hand, in step S201, if the number of image sensors is not one (step S201; NO), the image processing unit 34 sets the cycle value for repeating real-time processing to 1/30 seconds (step S204). In this case, the image processing unit 34 sets the cycle value read out by the processing of step S73 described above. By setting the cycle value to 1/30 seconds, the cycle value is set to a large value compared to one input, and it is possible to prevent the image synthesis processing described below from being unable to keep up.

Next, the connection unit 38 acquires two sets of captured image data from the imaging unit 40 (step S205). The two sets of captured image data are data of hemispherical images as shown in Figs. 4(a) and (b), respectively. In this case, the connection unit 38 executes processing using the image acquisition program (ProgC02 (two sets of images)) shown in Fig. 15(a).

Then, the image processing unit 34 synthesizes the two captured image data to create an equirectangular projection image EC as shown in FIG. 4(c) (step S206). In this case, the image processing unit 34 executes the processing according to the synthesis processing program (ProgS02 (synthesis processing)) shown in FIG. 15(b).

This completes the object detection processing in display area management in step S44.

Then, the communication unit 48 of the real-time data processing terminal 3 outputs the captured image data to the communication unit 58 of the near-terminal data processing device 5 (step S45). As a result, the communication unit 58 of the near-terminal data processing device 5 inputs the captured image data. Then, the storage/reading unit 59 of the near-terminal data processing device 5 reads the specified area information (center point CP and angle of view α) from the storage unit 5000 (step S46).

Next, the transmitting/receiving unit 51 of the near-terminal data processing device 5 transmits to the distributed data processing terminal 6 the photographed image data received in step S45 and the information indicating the center point CP and the angle of view α read in step S46 (step S47). As a result, the transmitting/receiving unit 61 of the distributed data processing terminal 6 receives the photographed image data and the information indicating the center point CP and the angle of view α.

Next, the display control unit 67 of the distributed data processing terminal 6 displays a display area setting screen as shown in FIG. 30 on the display 517 based on the captured image data received in step S47 (step S48). FIG. 30 is a diagram showing the display area setting screen. As shown in FIG. 30, the display area setting screen displays a detection area display screen d1 on the left side and an area setting screen d2 on the right side. Here, the display 517 is an example of a display means, and the display control unit 67 is an example of a display control means that displays a screen including multiple areas on the display 517. FIG. 31(a) is a diagram showing the position of the detection area in an actual store, and FIG. 31(b) is a diagram showing the relationship between the position of the detection area in the store and the specified area information.

Furthermore, the detection area display screen d1 in FIG. 30 includes multiple (four in this case) detection area display fields d11 to d14. The detection area display fields d11 to d14 display four predetermined area images Q11 to Q14, which are indicated by four predetermined areas T1 to T4, from one omnidirectional image (an example of a captured image) obtained by capturing an image of the inside of the store.

In addition, the area setting screen d2 has an attention area setting field d20 displayed in the center. The movement area setting fields d21 to d24 are displayed on the left, top, right, and bottom of this attention area setting field d20, respectively. The user inputs the numbers "1" to "4" displayed in the lower right of each detection area display field d11 to d14 into each field d20 to d24. The reception unit 62 then receives the input and sets the display position of the movement area relative to the attention area. Here, the detection area display field d11 is set in the attention area setting field d20. In addition, the detection area display fields d12, d13, and d14 are set in the movement area setting fields d21, d22, and d23, respectively. As a result, the display control unit 67 uses the specified area image Q11 displayed in the detection area display field d11 as a reference and displays the area a2 to the left of the area a1 in the store displayed in the specified area image Q11 in the detection area display field d12. The display control unit 67 also uses the predetermined area image Q11 displayed in the detection area display field d11 as a reference and causes the detection area display field d13 to display an area a3 at the back of the area a1 in the store displayed in the predetermined area image Q11. Furthermore, the display control unit 67 uses the predetermined area image Q11 displayed in the detection area display field d11 as a reference and causes the detection area display field d14 to display an area a4 to the right of the area a1 in the store displayed in the predetermined area image Q11.

Now, with reference to FIG. 32, the process of changing the area within the store displayed in the detection area display field will be described. FIG. 32 is a diagram showing a detection range setting screen. When a user selects, by tapping or the like, a predetermined area image Q11 displayed in the predetermined detection area display field (here, detection area display field d11) shown in FIG. 30, the reception unit 62 receives the selection. This causes the display control unit 67 to display a display area setting screen such as that shown in FIG. 32 on the display 515 of the distributed data processing terminal 6.

As shown in FIG. 32, the detection range setting screen displays the detectable range P, a setting button d8, and a cancel button d9. In the detectable range P, the specified area image Q11 of the selected detection area display field d11 is actually displayed.

The detectable range P is the range within which the imaging range of the imaging unit 40 can detect an object (such as a face). This detectable range P includes a matchable range C in which the image of the object can be matched, and an unmatchable range R outside of the matchable range C in which matching is not possible. In FIG. 32, margins a and b are predetermined values provided at the left and right ends in the x direction and the top and bottom ends in the y direction, respectively, of the detectable range P, and are determined in advance.

The reason why the matchable range C is provided separately from the detectable range P is that since the detectable range P is fixed as the imaging range of the imaging element 401a, etc., the matchable range C can be changed by changing the margin a and margin b, and an object can be detected within a range that matches the size of any room or area. In addition, this means that if an object enters the detectable range P and then leaves the detectable range P before entering the matchable range C, the real-time data processing terminal 3a does not need to transmit the matched data to the centralized data processing server 7 via the near-terminal data processing device 5 and the distributed data processing terminal 6, thereby reducing unnecessary communication charges.

The "Set" button d8 is a button that the user presses when deciding on the detection area. The "Cancel" button d9 is a button that the user presses when deciding on the detection area and wish to stop deciding on the detection area.

The detectable range P indicates the same range as the specified area T (see FIG. 7). Therefore, setting the detectable range P is equivalent to setting the specified area T. If the aspect ratios of the detectable range P and the specified area T are different, the display control unit 67 displays the specified area T in accordance with the size of the detectable range P by enlarging or reducing it while maintaining the aspect ratio of the specified area T. For example, if the aspect ratio of the detectable range P is 9:16 and the aspect ratio of the specified area T is 3:4, the display control unit 67 displays the specified area T so that its width is maximized within the detectable range P. This detectable range P can be changed by the user at the distributed data processing terminal 6. This change and setting will be described using FIG. 29.

First, the reception unit 62 of the distributed data processing terminal 6 receives input from the user via the touch panel 521 or the like (S51). The types of input include changing the image of the predetermined area within the detectable range P by swiping, pinching in or out, pressing the "Set" button d8, or pressing the "Cancel" button d9 on the detection range setting screen shown in FIG. 32. Changes to the detectable range P include, in FIG. 7(a), displaying a different predetermined area within the three-dimensional sphere CS by swiping (movement), displaying a wide range of predetermined areas including the predetermined area T by pinching in (reducing), and displaying a narrow range of predetermined areas within the predetermined area T by pinching out (enlarging).

Here, an example of changing the predetermined area image displayed in the detectable range P will be described with reference to Figure 33. Figure 33(a) is a diagram showing the predetermined area image Q11 before change in the detectable range P, Figure 33(b) is a diagram showing the predetermined area image Q111 after change in the detectable range P, and Figure 33(c) is a diagram showing an example of a warning display for the predetermined area image Q112 after change in the detectable range P.

As shown in FIG. 33(a), when the user swipes their hand h1 to the right, the reception unit 62 receives a change to the detectable range P. This causes the display control unit 67 to change the specified area image Q11 displayed in the detection area display field d11 to the specified area image Q111. Then, when the user presses the "Set" button d8, the reception unit 62 receives the setting, and the display control unit 67 displays the display area setting screen shown in FIG. 30. In this case, the specified area screen Q111 shown in FIG. 33(b) is displayed in the detection area display field d11.

Furthermore, if the user swipes their hand h1 to the right in the state shown in FIG. 33(b), causing the display control unit 67 to display an image (predetermined area image Q112) that is the same as the predetermined area image Q12 on the detectable range P, the positional relationship between each of the predetermined area images Q11 to Q14 on the detection area display screen d1 shown in FIG. 30 and each of the area numbers "1" to "4" on the area setting screen d2 will no longer match. Therefore, as shown in FIG. 33(c), the display control unit 67 displays a warning by superimposing a masking image (for example, an image with a semi-transparent red background color) on the predetermined area image Q112.

Here, the processing performed by the display control unit 67 when displaying a warning will be described in more detail. Figure 31(b) is a diagram showing the relationship between the position of the detection area in the store and the specified area information. If the virtual camera IC shown in Figure 8 is placed at the center of circle C and the elevation angle "ea" in Figure 8 is fixed, the center point CP of each area can be defined by the azimuth angle "aa" as shown in Figure 31(b). Note that in Figure 31(b), the center points for areas a1 and a2 in the store are indicated by ● for convenience.

In this embodiment, as an example, a method will be described in which the display control unit 67 that displays a warning compares the azimuth angles of the attention area a1 and the moving areas a2, a3, and a4 to determine whether the arrangement conditions set by the user on the area setting screen in FIG. 30 are satisfied. For example, the azimuth angle "aa1" of the area a1 in the store shown in FIG. 31(b) is set as the starting point (0 degrees), and the angle increases counterclockwise. If the arrangement relationship of the areas a1, a2, a3, and a4 in the store set by the user is as shown in FIG. 31(b), and the azimuth angles of the areas a2, a3, and a4 in the store are aa2, aa3, and aa4, respectively, the condition that satisfies this arrangement is aa1 < aa2 < aa3 < aa4. As in the case of FIG. 31(b), if the above-mentioned conditions are not satisfied when the specified image of any of the moving areas a2, a3, and a4 is changed, the display control unit 67 displays a warning as shown in FIG. 33(c).

The judgment by the display control unit 67 when to display a warning is not limited to the above example. For example, instead of setting the condition for satisfying the arrangement as aa1 < aa2 < aa3 < aa4, a predetermined threshold value for the horizontal angle aa may be set for each of the regions a1, a2, a3, and a4, and the judgment as to whether or not to display a warning may be made depending on whether or not each threshold value is exceeded. Furthermore, the above-mentioned threshold value may be set as a common value for each of the regions a1, a2, a3, and a4, and the judgment method (judgment condition) may be changed to process whether or not to display a warning.

Next, returning to FIG. 29, if the type of input is a change in the detectable range P (S52; change), the display control unit 67 changes and displays the detectable range P according to the amount of pinch-in or pinch-out operation (S53). Then, the process returns to step S51.

Also, if the type of input is confirmation of the detection area by pressing the "Set" button d8 (S52; Confirm), the transmitter/receiver 61 transmits a confirmation request to the near-terminal data processing device 5 (S54). This confirmation request includes the specified area information indicating the center point CP and the angle of view α for identifying the detectable range P (specified area T) confirmed in steps S51 and S52. Note that, if the detectable range P (specified area T) is confirmed after being changed in steps S51 and S52, the specified area information indicating the center point CP and the angle of view α after the change is included. As a result, the transmitter/receiver 51 of the near-terminal data processing device 5 receives the confirmation request.

Next, in the near-terminal data processing device 5, the memory/read unit 59 stores the determined specific area information in the memory unit 5000 (S55).

Also, when the input type is the pressing of the "Cancel" button d9 to cancel the setting of the detection area (S52; Cancel), the transmission/reception unit 61 transmits a request to stop shooting to the near-terminal data processing device 5 (S56). As a result, the transmission/reception unit 51 of the near-terminal data processing device 5 receives the request to stop shooting. Then, the communication unit 58 of the near-terminal data processing device 5 outputs the request to stop shooting to the communication unit 48 of the real-time data processing terminal 3 (S57). As a result, the communication unit 48 of the real-time data processing terminal 3 inputs the request to stop shooting. Then, in the real-time data processing terminal 3, the connection unit 38 causes the imaging unit 40 to stop shooting (S58). After the processing of step S56, the display control unit 67 returns to the display of the detection range setting screen of FIG. 30 (S59).

This completes the process of setting the detection area.

<Preparation for image recognition>
Next, a preparation process for image recognition will be described with reference to Fig. 34 to Fig. 36. Fig. 34 is a sequence diagram showing a process for requesting to start image recognition.

As shown in FIG. 34, in the distributed data processing terminal 6, the reception unit 62 receives a request to start image recognition from a user (step S61). In this case, the GUI (Graphical User Interface) of the distributed data processing terminal 6 is used. As a result, the transmission/reception unit 61 of the distributed data processing terminal 6 transmits image recognition start request information indicating a request to start image recognition to the near-terminal data processing device 5 (step S62). This start request information includes an authentication server ID for identifying the authentication server 9 that performed the authentication in the above-mentioned authentication process (see FIG. 24 and FIG. 25). As a result, the transmission/reception unit 51 of the near-terminal data processing device 5 receives the image recognition start request information. Then, the communication unit 58 transmits image recognition start request information to the real-time data processing terminal 3 (step S63). As a result, the communication unit 48 of the real-time data processing terminal 3 receives the image recognition start request information. In this way, by separating the user interface from the real-time data processing terminal 3, remote operation from the distributed data processing terminal 6 is possible.

FIG. 35 is a sequence diagram showing preparation processing for real-time processing by the real-time data processing terminal.
35, the connection unit 38 of the real-time data processing terminal 3 acquires the model number of the imaging unit 40 from the imaging unit 40 (step S71). In this case, the connection unit 38 requests the imaging unit 40 for the model number, and the imaging unit 40 transmits its own model number in response to this request.

Next, the storage/readout unit 39 searches the imaging element information management DB 3001 (see FIG. 14(a)) using the model number acquired in step S71 as a search key to read out the corresponding number of imaging elements and the type of lens (step S72). Furthermore, the storage/readout unit 39 searches the cycle value management DB 3002 (see FIG. 14(b)) using the number of imaging elements read in step S72 as a search key to read out the corresponding cycle value (step S73).

Next, the storage/readout unit 39 searches for an image acquisition program managed in the image acquisition program management DB 3003 (see FIG. 15(a)) using the number of image sensors read out in step S72 as a search key (step S74). Next, the storage/readout unit 39 searches for a synthesis processing program managed in the synthesis processing program management DB 3004 (see FIG. 15(b)) using the number of image sensors read out in step S72 as a search key (step S75). Next, the storage/readout unit 39 searches for a distortion correction program managed in the distortion correction program management DB 3005 (see FIG. 15(c)) using the type of lens read out in step S72 as a search key (step S76). Next, the storage/readout unit 39 searches for a service program managed in the service program management DB 3006 (see FIG. 15(d)) using the type of lens read out in step S72 as a search key (step S77).

Next, the judgment unit 33 judges whether all of the programs to be executed have been installed based on the search results of steps S74 to S77, depending on whether the programs to be executed are managed (step S78). For example, in step S74, the storage/readout unit 39 searches for an image processing program, and if the image processing program is managed, the judgment unit 33 judges that the program has been installed, and if the image processing program is not managed, the judgment unit 33 judges that the program has not been installed.

If the judgment unit 33 judges that all four programs are managed (step S78; YES), the process shown in FIG. 35 ends. On the other hand, if the judgment unit 33 judges that at least one of the four programs is not installed (step S78; NO), the process proceeds to step S91 described below.

Figure 36 is a sequence diagram showing the program acquisition process. This shows the process in which the real-time data processing terminal 3 acquires from the service providing server 8 a program that has been found not to be installed by the process shown in Figure 35.

First, as shown in FIG. 36, the communication unit 48 of the real-time data processing terminal 3 transmits request information indicating a request for a program that is not installed to the communication unit 58 of the near-terminal data processing device 5 (step S91). This request information includes the name of the requested program.

Next, the transmitting/receiving unit 51 of the near-terminal data processing device 5 transmits the request information received by the communication unit 58 to the transmitting/receiving unit 61 of the distributed data processing terminal 6 (step S92). Then, the transmitting/receiving unit 61 of the distributed data processing terminal 6 transmits the request information to the transmitting/receiving unit 81 of the service providing server 8 (step S93).

Next, in the service providing server 8, the storage/reading unit 89 reads out the program indicated by the program name included in the request information (step S94). Then, the transmitting/receiving unit 81 transmits the read out program to the transmitting/receiving unit 61 of the distributed data processing terminal 6 (step S95). At this time, the program name is also transmitted.

Next, the transmitting/receiving unit 61 of the distributed data processing terminal 6 transmits the program including the program name to the transmitting/receiving unit 51 of the near-terminal data processing device 5 (step S96). Then, the communication unit 58 of the near-terminal data processing device 5 transmits the program including the program name to the communication unit 48 of the real-time data processing terminal 3 (step S97).

Next, the storage/read unit 39 of the real-time data processing terminal 3 installs the program acquired by the communication unit 48 and manages the names of the programs to be installed from among the tables shown in FIG. 15 (step S98).

Next, the storage/readout unit 39 starts all programs necessary for the image recognition process (step S99). As a result, the real-time data processing terminal 3 executes the started programs, thereby starting the real-time processing shown below.

<Image recognition processing>
Here, a case will be described in which the "Login to object detection processing service" button b1 shown in Fig. 26 is pressed and the service providing server 8 executes the object detection processing service among the image recognition processing services. Fig. 37 is a sequence diagram showing the image recognition processing. First, the real-time data processing terminal 3 performs real-time processing (step S111).

(Object detection processing)
Here, the object detection process among the real-time processes will be described with reference to Fig. 38(a) and (b). Note that, here, the process of Fig. 38(b) is performed after the process of Fig. 38(a). However, since the process of Fig. 38(a) has already been described, the process of Fig. 38(b) will be described.

After the process of step S203, the object detection unit 35 detects feature points that are candidates for the object in the captured image data, thereby detecting the object (step S207). In this case, the image processing unit 34 executes the process by the service program (ProgD01 (object detection process)) shown in FIG. 15(d). The image processing unit 34 searches for a rectangle from the edge of the captured image while comparing it with shape model data of the object stored in advance in the storage unit 3000, and selects the position of the most likely feature point of the object. Note that the process of step S207 can use known technology, such as that disclosed in "Face Recognition Technology and Its Applications: Special Issue on Elemental Technologies and Solutions Supporting Public Safety; Biometrics Authentication, NEC Technical Journal, Vol. 63, no. 3, pp. 26-30, 2010-09".

Next, the image processing unit 34 corrects distortion of the image of the detected object (step S208).
In this case, the image processing unit 34 executes the process according to the image processing program (ProgW01 (wide-angle distortion correction)) shown in FIG. 15(c).

After the processing of step S206, the process proceeds to step S207, where the object detection unit 35 detects object candidates in the data of the equirectangular projection image EC, thereby detecting the object. Next, in step S208, the image processing unit 34 corrects distortion of the image of the detected object. In this case, the image processing unit 34 executes processing using the image processing program (ProgW02 (fisheye distortion correction)) shown in FIG. 15(c).

As a result, as shown in FIG. 37, the communication unit 48 of the real-time data processing terminal 3 transmits the captured image data to the communication unit 58 of the near-terminal data processing device 5 (step S112). Then, the transmission/reception unit 51 of the near-terminal data processing device 5 transmits the captured image data received in step S112 to the transmission/reception unit 61 of the distributed data processing terminal 6 (step S113). As a result, the display control unit 67 of the distributed data processing terminal 6 displays the captured image shown in FIG. 45 on the display 517 in real time (step S114). FIG. 45 is a diagram showing an example of a display of a captured image in the distributed data processing terminal. In this captured image, a frame indicating a rectangle where an object (here, a face) has been detected is displayed. Steps S112 to S114 are streaming processing.

(Event generation process)
Next, the event generation process of the real-time processing will be described with reference to Figures 39 and 40. Figures 39 and 40 are flow charts showing the event generation process of the real-time processing.

If the object detection unit 35 does not detect an object (here, a face) as a result of the processing in step S207 (S211; NO), the process returns to the processing in step S211 after waiting for a predetermined time (set cycle value) (S212). For example, detection is performed 30 times per second.

In contrast, if the object detection unit 35 detects an object (here, a face) by the processing of step S207 described above (S211; YES), the process proceeds to step S213. Also, if the position of the detected object is not within the collation range C (outside the collation range C) (S211; NO), the process proceeds to step S212.

In step S213, the storage/readout unit 39 stores the object information about the object detected in step S207 as a new record in the object information management table (see FIG. 16). In this case, the storage/readout unit 39 erases the end flag for the record already stored in the object information management table to reset it.

Next, the storage/read unit 39 erases the end flags for all records in the object displacement management table in order to initialize the end flags stored in the object displacement management table (S214). Note that the end flags are flags that are added in step S216, which will be described later.

Next, if there are records remaining to be read in the object displacement management table (S215; YES), the memory/read unit 39 reads data for one record from the object displacement management table (S216). On the other hand, if all records have been read (S215; NO), the process proceeds to step S220, which will be described later.

Next, the calculation unit 31 calculates the distance between the latest (current) object detection position stored in step S213 and the previous object detection position (here, coordinates (x, y)) stored in step S216 (S217). Then, the judgment unit 33 judges whether the distance calculated in step S217 is shorter (less than) a predetermined threshold (S218). In this embodiment, the calculation is performed using the squared Euclidean distance shown in the following (Equation 2). Note that the judgment unit 33 may also judge whether it is equal to or less than the predetermined threshold.

The squared Euclidean distance has the advantage that it is easier to find the position of past objects that are closer because the more distant the object, the greater the weight. However, other well-known methods of calculating distance can also be used, such as counting the city block distance or the number of pixels that are shifted in only a specific direction (x or y direction).

Next, if the distance calculated in step S217 is shorter than a predetermined threshold (S218; YES), the distance between the current object position and the past positions of the remaining records managed in the object displacement management table is not calculated, and the process proceeds to step S219. If the distance calculated in step S217 is not shorter than (the same as or longer than) the predetermined threshold (S218; NO), the process returns to step S215.

Next, the memory/read unit 39 updates the object displacement management table by replacing the position information indicating the past object position whose distance was determined to be shorter than the threshold with the position information indicating the latest object position whose distance was determined to be shorter than the threshold (S219). In this case, the memory/read unit 39 adds an "end flag" to the end flag column of the record that includes the position information used for the replacement in the object information management table and the object displacement management table.

Next, if there are any records remaining in the object displacement management table that do not have an end flag set (S220; YES), the communication unit 48 of the real-time data processing terminal 3 outputs a display change request to the communication unit 58 of the near-terminal data processing device 5, indicating a request to change the display (S211).

Here, the process of displaying the movement count number and storing the movement information will be explained using FIG. 41. FIG. 41 is a sequence diagram showing the process of displaying the entry/exit count number and storing the movement information.

First, the communication unit 48 of the real-time data processing terminal 3 transmits a display change request to the communication unit 58 of the near-terminal data processing device 5 in step S221 above, indicating a request to change the display. This display change request includes an area number indicating the area in the store where the object as a person was present, and an object ID. As a result, the communication unit 58 of the near-terminal data processing device 5 inputs the display change request. Then, the transmission/reception unit 51 transmits (transfers) the display change request to the distributed data processing terminal 6 (S302). As a result, the transmission/reception unit 61 of the distributed data processing terminal 6 receives the display change request.

Next, the distributed data processing terminal 6 performs a change display process (S303). Here, the process of step S303 will be explained in detail with reference to FIG. 42.

The storage/reading unit 69 of the distributed data processing terminal 6 searches the object presence management DB 6001 (see the object presence management table in FIG. 18(a)) using the object ID received in step S302 as a search key (S303-1). Then, the judgment unit 63 judges whether an object ID identical to the object ID of the search key is stored in the object presence management DB 6001 (S303-2).

If the object ID is not stored in the object presence management table (S303-2; NO), the storage/reading unit 69 further reads out the number of the "area of interest" from the display area management DB 6002 (see the display area management table in Figure 18 (b)) (S303-3).

Next, the judgment unit 63 judges whether the area number received in step S302 matches the number of the attention area read out in step S303-3 (S303-4). If they do not match (S303-4; NO), the processing of FIG. 42 ends. On the other hand, if they match (S303-4; YES), a new record is added to the object presence management table of the object presence management DB 6001, and the received person number is stored (S303-5). Note that in this embodiment, the area number "1" of the entrance of the store, etc. is set as the attention area, but a different area number can be set as the attention area by storing a display area management table according to the layout of the store, etc., or the area to be counted. After the processing of step S303-5, the processing of FIG. 42 ends.

Next, in step S303-2 above, if the object ID is stored in the object presence management table (S303-2; YES), the storage/reading unit 69 reads out the number of the area of interest from the display area management DB 6002 (S303-6).

Next, the judgment unit 63 judges whether the area number received in step S302 is different from the number of the attention area read out in step S303-6 (S303-7). If the area number is not different (the same) as the number of the attention area (S303-7; NO), the processing of FIG. 42 ends. Note that the movement of a person when the area number and the attention area number match is the movement when the person once leaves the area a1 in the store indicated by the attention area "1", but returns to the area a1 again without moving to another area a2 in the store indicated in FIG. 31. Note that in this embodiment, the attention area "1" described above is treated as the first area identification information for identifying the first area, and the other attention area numbers are treated as the second area identification information for identifying the second area.

On the other hand, if the area number is different from the number of the attention area (S303-7; YES), the storage/readout unit 69 searches the display area management DB 6002 using the area number received in step S302 as a search key to read out the corresponding movement area information (left, top, right, bottom) (S303-8). Then, if the movement area is "left" (S303-9; left), the calculation unit 65 counts up to indicate a movement to the left (S303-10). Also, if the movement area is "up" (S303-9; up), the calculation unit 65 counts up to indicate a movement to the top (S303-11). Furthermore, if the movement area is "right" (S303-9; right), the calculation unit 65 counts up to indicate a movement to the right (S303-12). Note that in this example, the area below the attention area is outside the store, so it is not counted up.

Next, the display control unit 67 changes the display of the count display area m3 in the upper right portion of the matching result display screen as shown in FIG. 45 (S303-13). In this example, area number 1 is set as the reference (start area), and it is displayed that up to now, one person has moved to the left area a2, two people have moved to the upper (back) area a3, and three people have moved to the right. Note that the "Display Movement Information" button e1 displayed in the lower right of the matching result display screen is a button for displaying the number of counts per unit time, and will be described later.

This completes the processing of step S303 (see Figure 42).

Returning to FIG. 41, the transmission/reception unit 61 of the distributed data processing terminal 6 then transmits an object position storage request indicating a request to store the position of an object such as a person to the centralized data processing server 7 (S304). In addition to the area number and object ID received in step S302, this object position storage request also includes information on the date and time when the distributed data processing terminal transmits the object position storage request. As a result, the transmission/reception unit 71 of the centralized data processing server 7 receives the object position storage request.

Next, in the centralized data processing server 7, the storage/readout unit 79 associates the information indicating the date and time, area number, and object ID received in step S304 with each other and stores them as one record in the object position management DB 7002 (see FIG. 19(b)) (S305).

In addition, a record that associates a date, time, area number, and object ID may be deleted after a certain period of time (e.g., three months) has passed since it was stored.

Returning to FIG. 39, after processing of step S221, the storage/readout unit 39 deletes the record that does not have the end flag set (S222) and proceeds to processing of step S231 shown in FIG. 40. This processing is for deleting the record from the object displacement management table and excluding it from future comparison targets, since if the record remains in the object displacement management table without the end flag set, it can be assumed that the object (here, a person) has already moved out of the collation possible range C described below.

On the other hand, if there are no records remaining in the object displacement management table that do not have an end flag (S220; NO), the process of step S221 is skipped and the process proceeds to step S231.

Next, if any records to which no end flag has been added remain in the object information management table (S231; YES), the storage and reading unit 39 adds the position information of these remaining records as new records to the object displacement management table (S232). In this case, the storage and reading unit 39 manages the matching progress status of the added record as "0 (before recognition)". This process is for managing the subsequent displacement, since it can be assumed that a new object (here, a person) has entered the detectable range P described below.

In this case, the storage and readout unit 39 stores the previously added ID value in the ID column of the added record by incrementing it by 1. Note that the storage and readout unit 39 does not renumber the IDs when deleting a record in step S221. In this way, it is possible to store the correct result for a record whose matching progress status is 1 (matching in progress).

On the other hand, if there are no records remaining in the object information management table to which an end flag has not been added (S231; NO), the process proceeds to step S236, which will be described later.

Next, after processing in step S232, the judgment unit 33 judges whether an object with a matching progress status of "0 (before matching)" in the object displacement management table is within the matching range C (S233). This judgment is made based on the position information of the object. Then, if it is determined in step S233 that the object is within the matching range C (S233; YES), the image processing unit 34 refers to the position, width, and height of the object within the matching range C in the object displacement management table, cuts out partial image data determined by these values, and encodes it in a general-purpose format such as JPEG (S234). In this case, the storage/reading unit 39 changes the matching progress status of the corresponding record in the object displacement management table to "1 (matching in progress)."

On the other hand, if it is determined in step S233 that the match is not within the matching range C (S233; NO), the process proceeds to step S236.

Next, the event generation unit 36 generates an event message to notify the near-terminal data processing device 5 of the transmission of the partial image data and the image ID of the partial image associated with the partial image data (S235). Specifically, the event generation unit 36 generates an event message such as "Send."

Next, the image processing unit 34 clears the displayed text image, and then references the position associated with the matching progress status of "1 (matching in progress)" or "2 (matching completed)" in the object displacement management table, and composites the text image at the corresponding position in the captured image (S236).

Here, FIG. 45 shows an example of combining text with an image. FIG. 45 shows the range of photography by the imaging unit when an image acquisition terminal is installed in a room (here, the entrance of a building). The image processing unit 34 combines a standby state mark m1 represented by "!" with the partial image at position (x2, y2) to indicate that matching is in progress. The image processing unit 34 also combines object ID information m2 represented by "Kato" with the corresponding position of the partial image by referring to the name to indicate the matching result with respect to the partial image at position (x3, y3). This object ID information m2 is information read out from the object displacement management DB 3008. The combined image data is transmitted to the distributed data processing terminal 6 by steps S112 and S113 shown in FIG. 37, and is displayed in step S114.

Next, if the communication unit 48 has not received the matching result from step S318 described below (S237; NO), the process proceeds to step S212.

On the other hand, when the communication unit 48 receives the matching result (object ID and image ID of the partial image) (S237; YES), the storage/readout unit 39 stores the received name in the name field associated with the image ID of the received partial image in the object displacement management table (S238). In this case, the storage/readout unit 39 changes the matching progress status from "1 (matching in progress)" to "2 (matching completed)". Then, the image processing unit 34 clears the displayed character image, and then refers to the position in the object displacement management table associated with the image ID of the received partial image, and synthesizes the character image with the received name at the corresponding position in the captured image (S239). Then, the process returns to step S212. In step S239, the display control unit 37 updates the display of "!" that was being matched to a display of the name, such as "Kato". This allows the user of the distributed data processing terminal 6 to quickly know the progress status of the matching.

This completes the event generation process, which is part of real-time processing.

Next, returning to FIG. 37, the communication unit 48 of the real-time data processing terminal 3a transmits the event message, partial image data of the object (here, a face), and the image ID of the partial image to the communication unit 58 of the near-terminal data processing device 5 (S115).

Next, in the near-terminal data processing device 5, the data detection unit 56 detects whether or not a "Send" event message has been received by the communication unit 58 (S116). Then, if reception of an event message is detected (S116; YES), the communication unit 58 receives the partial image data and the image ID of the partial image that were sent together with the event message (S117). Then, the storage/readout unit 59 temporarily stores the partial image data and the image ID of the partial image in the storage unit 5000 (S118).

Next, the data detection unit 56 monitors whether the reception of the partial image data is complete (S119). This process of step S119 is repeated until all partial image data and image IDs of partial images are received for one event message (S119; NO). Then, when the reception of the partial image data is complete (S119; YES), the storage/readout unit 59 reads out all partial image data and image IDs of partial images sent from the storage unit 5000 together with one event message and temporarily stored in the storage unit 5000 (step S120). After that, the transmission/reception unit 51 transmits all partial image data and image IDs of partial images read out in step S120 to the transmission/reception unit 61 of the distributed data processing terminal 6 via the intranet 200 (S121). As a result, the transmission/reception unit 61 of the distributed data processing terminal 6 receives all partial image data and image IDs of partial images. The partial image data is used as matched data later.

(Matching process)
Next, the matching process will be described with reference to Fig. 43 to Fig. 45. Fig. 43 is a sequence diagram showing the matching process. Fig. 44 is a flowchart showing the matching process. Fig. 45 is a diagram showing the shooting range of an imaging unit when an image acquisition terminal is installed in a room.

First, when the transmission/reception unit 61 of the distributed data processing terminal 6 receives the partial image data as the data to be matched and an object ID (object ID related to the partial image) for identifying the object shown in the partial image through the processing of step S121 described above, the transmission/reception unit 61 transmits matching request information indicating a matching request to the centralized data processing server 7 (S313). This matching request information includes the object ID related to the data to be matched and the partial image. As a result, the transmission/reception unit 71 of the centralized data processing server 7 receives the matching request information.

Next, the centralized data processing server 7 performs a matching process (S314). Here, the matching process is explained using FIG. 44.

As shown in FIG. 44, the feature generating unit 74 of the centralized data processing server 7 converts the data to be matched received in step S313 to decode it into bitmap data, and generates feature parameters for identifying an individual, such as the contours and inclination of the eyes and nose of the facial image related to the data to be matched (S401).

Next, the memory/read unit 79 searches the matching data management DB 7001 to determine whether any registered matching data remains (S402). If it is determined that any matching data remains, the matching unit 75 compares the feature parameters of both data (matching data and matched data) to calculate the similarity (S403). The memory/read unit 79 then temporarily stores the object ID attached to the matching data and the "similarity" calculated in step S403 as a pair in the memory unit 7000 (S404). After that, the matching data is changed to the next row in the matching data management table shown in FIG. 19(a), and the process of step S402 is performed again.

On the other hand, if it is determined in step S402 that no matching data remains (including the case where there is no matching data at all), the process proceeds to step S405. Then, the determination unit 73 determines whether the maximum similarity among the similarities temporarily stored in the storage unit 7000 is greater than a threshold value (S405).

If the determination unit 73 determines in step S405 that the maximum similarity is greater than the threshold value (YES), the storage/reading unit 79 reads out the object ID attached to the matching data with the maximum similarity from the matching data management DB 7001 (S406). The threshold value is, for example, "80%." As a result, if the similarity is low, the object ID is not read out.

On the other hand, in step S405, if the judgment unit 73 judges that the maximum similarity among the similarities temporarily stored in the memory unit 7000 is equal to or less than the threshold value (NO), the memory and readout unit 79 registers the data to be matched as matched data (S407). If matched data has already been registered, the memory and readout unit 79 sets "the current number of matched data + 1" as an object ID and registers the matched data in association with this object ID. Next, the feature generation unit 74 generates features of the registered matched data (S408). Then, the memory and readout unit 79 reads out the object ID associated in step S407 (S409).

This completes the process shown in Figure 44.

Returning to FIG. 43, the transmission/reception unit 71 of the centralized data processing server 7 transmits matching result information indicating the matching result to the distributed data processing terminal 6 (S315). This matching result information includes the object ID read in step S406 or step S409 or the information read in step S407, and the object ID related to the partial image received in step S313. As a result, the transmission/reception unit 61 of the distributed data processing terminal 6 receives the matching result information.

Next, in the distributed data processing terminal 6, the display control unit 67 displays the matching result as shown in FIG. 45 (S316). Here, the display of the matching result will be specifically explained using FIG. 45. FIG. 45 is a diagram showing the matching result display screen.

As with the detection range setting screen shown in FIG. 32, the detectable range P on the matching result display screen is the range within which the shooting range of the imaging unit 40 can detect an object (here, a face). This detectable range P includes a matchable range C in which the image of the object (here, a face) can be matched, and an unmatchable range R outside this matchable range C in which matching is not possible. In FIG. 45, margins a and b are predetermined values set at the left and right ends in the x direction and the top and bottom ends in the y direction, respectively, of the detectable range P. These values are predetermined.

The reason why the matchable range C is provided separately from the detectable range P is that the detectable range P is fixed as the imaging range of the imaging element 401a, etc., so that an object can be detected within a range that matches the size of any room or area. This also means that if an object enters the detectable range P and then leaves the detectable range P before entering the matchable range C, the real-time data processing terminal 3a does not need to transmit the matched data to the centralized data processing server 7 via the near-terminal data processing device 5 and the distributed data processing terminal 6, thereby reducing unnecessary communication charges.

The face detection by the process of step S207 described above is performed throughout the entire detectable range P, which is the shooting range. If the object is outside the rectangular range determined by the margins a and b, the object is not within the matchable range C. In this case, the object is detected within the detectable range P, but not within the matchable range C. As a result, the object is present in the unmatchable range R, which is the range outside the matchable range C within the detectable range P.

Here, (x1, y1) indicates the coordinates of the upper left corner of the partial image data when an object is detected within the unmatchable range R.

(x2, y2) indicates the coordinates of the upper left corner of the partial image data when an object is detected within the matching range C. Also, here, a standby state mark m1 represented by "!" is displayed. This standby state mark m1 is displayed from the distributed data processing terminal 6 sending the matching request information (S313) until it receives the matching result information from the centralized data processing server 7 (S315).

(x3, y3) indicates the coordinates of the upper left corner of the partial image data when an object present within the matchable range C is detected. Also displayed here is object ID information m2 represented by "S001". This object ID information is the "partial image object ID" received from the centralized data processing server 7 in step S315. In this way, when an object enters the matchable range C, a standby state mark m1 is initially displayed, but after 1 to 2 seconds, object ID information m2 is displayed. This allows the viewer to recognize the displayed object (here, a face).

Furthermore, as described above, the count display area m3 is displayed in the upper right corner of the matching result display screen. This makes it easy to see that, in FIG. 31, one person who entered the store from the entrance moved from area a1 to area a2, two people moved from area a1 to area a3, and three people moved from area a1 to area a4.

Returning to FIG. 43, the transmitting/receiving unit 61 of the distributed data processing terminal 6 transmits (transfers) the matching result to the near-terminal data processing device 5 (S317). This matching result contains the same information as in step S315 above. As a result, the transmitting/receiving unit 51 of the near-terminal data processing device 5 receives the matching result.

Next, the communication unit 58 of the near-terminal data processing device 5 transmits (transfers) the matching result to the communication unit 48 of the real-time data processing terminal 3 (S318). This matching result contains the same information as in step S317 above. As a result, the communication unit 48 of the real-time data processing terminal 3 receives the matching result.

(Display of movement information)
Next, the process of displaying movement information will be described with reference to Fig. 46 to Fig. 48. Fig. 46 is a sequence diagram showing the process of displaying movement information. Fig. 47 is a flow chart showing the process of counting movements to a predetermined area in a building (e.g., areas a1 to a4 in Fig. 31) taking into account the moving direction of an object. Fig. 48 is a diagram showing a movement result display screen.

First, when the user presses the "Display movement information" button e1, the reception unit 62 of the distributed data processing terminal 6 accepts the display of the movement information (S501). Then, the transmission/reception unit 61 transmits a movement information request to the centralized data processing server 7 to request the movement information (S502). This movement information includes the current date and time and a display area management table. As a result, the transmission/reception unit 71 of the centralized data processing server 7 receives the movement information request.

Next, the counting unit 76 of the centralized data processing server 7 counts the number of records by the combination of the area number indicating the area in the store to which the user has moved per unit time (per hour in this embodiment) and the object ID (S504). Here, the process of step S504 will be described in detail with reference to FIG. 47.

First, the storage/readout unit 79 refers to the area number column of the object position management table of the object position management DB 7002, and starts searching from the top row downwards for a record in which an area number matching the number in the attention area column of the display area management DB 601 received in step S502 is managed (S504-1). Then, if the date and time in the date and time column of the searched record is not within the read range (or is outside the range) (S504-2; NO), the processing of FIG. 47 ends. For example, if the current date and time received is April 18, 2019, the same "day" is within the range and a different "day" is outside the range. On the other hand, if the date and time in the searched record (an example of a first record) is within the read range (S504-2; YES), the storage/readout unit 79 reads the object ID of the searched record and searches for a record (an example of a second record) in which a matching object ID is managed in the subsequent records (S504-3). If there is a record that manages a matching object ID and the date and time managed in this record is within the read range (S504-4; YES), proceed to step S504-5. On the other hand, if there is no record that manages a matching object ID, or there is a record that manages a matching object ID but the date and time managed in this record is not within the read range (S504-4; NO), return to step S504-1.

Next, if there is a record that manages a matching object ID and the date and time managed in this record is within the read range (S504-4; YES), the storage and reading unit 79 searches the time zone column of the counting management DB 7003 based on the date and time managed in the record searched for in step S504-3 above, and identifies the record to be counted (S504-5). In this embodiment, there are time zone records every hour, and for example, if the time zone of the searched record is between 8:00 a.m., a record in which the time zone column managed in the counting management DB 7003 contains 2019-04-16 08 is identified.

Next, the storage/read unit 79 extracts the movement area information (left, up, right, down) of the object such as a person from the data in the display area management DB 6001 received in step S502 from the distributed data processing terminal 6, using the area number in the record searched in step S504-3 as a search key (S504-6). Then, if the movement area is "left" (S504-7; left), the counting unit 76 counts up ("+1") to indicate a movement to the left (S504-8). Also, if the movement area is "up" (S504-7; up), the counting unit 76 counts up ("+1") to indicate a movement to the up (S504-9). Furthermore, if the movement area is "right" (S504-7; right), the calculation unit 65 counts up ("+1") to indicate a movement to the right (S504-10). In this example, the area below the area of interest is outside the store, so it is not counted up. After steps S504-8, 9, and 10 are processed, the process returns to step S504-1 above.

This completes the process shown in Figure 47.

Returning to FIG. 46, the transmitter/receiver 71 of the centralized data processing server 7 transmits movement information to the distributed data processing terminal 6 (S505). This movement information includes count data, which is data resulting from the counting performed in step S504.

Next, in the distributed data processing terminal 6, the display control unit 67 displays the movement information received in step S505 on the display 517 as a movement result display screen represented by a bar graph as shown in Fig. 48 (S506). Fig. 48 shows an example of a display in which the counting results are graphed. Each bar graph shows the counting results for each direction stacked up.
By displaying the graph for each unit time in this way, it is possible to visualize in which direction the person has moved. In the example of FIG. 48, since it is the current day, the time after the time when the movement information display button e1 is pressed (for example, after 20:00) is not displayed. In addition, the arrow arranged to the right of the date at the top of the graph can display the results of the next day, but since it is the current day in FIG. 48, it is indicated by a light display that there is no reaction even if it is pressed. Here, if the arrow to the left of the date is pressed, the graph of the day before is updated and displayed according to the process of FIG. 46. In this case, the results of all 24 hours are displayed, and the arrow to the right of the date is displayed dark, and the results of the day can be redisplayed. In addition, when the reception unit 62 receives the pressing of the "OK" button f1 at the bottom right of FIG. 48, the display control unit 67 switches from the movement result display screen of FIG. 48 to the collation result display screen of FIG. 45.

[Main Effects of the Embodiments]
As described above, according to the present embodiment, by setting a predetermined area in a single omnidirectional image, the predetermined area image, which is an image of the predetermined area, can be set as a detectable range in the omnidirectional image in which an object can be detected. That is, the predetermined area image and the detectable range image are the same. This provides an effect that a desired predetermined area in a captured image can be easily set as a detectable range for an object.

The distributed processing system 100 also transmits partial image data, which is a region of a specific object, to the centralized data processing server 7 (S313), receives an object ID for identifying the assigned object from the centralized data processing server 7 as a result of the centralized data processing server 7 checking (S315), and transmits to the centralized data processing server 7 a first region number for identifying a first region among multiple regions included in the region captured by the single imaging unit 40 and the object ID of the object present in the first region in association with each other (S304). This provides the effect of being able to detect a specific object even if it is divided into multiple regions by a single imaging unit. Furthermore, by repeating the above-mentioned transmission, it provides the effect of being able to detect and track a specific object using a single imaging unit even if the specific object moves and the movement of the specific object spans multiple regions.

In addition, when the same object is photographed by the real-time data processing terminal 3, the photographed image data and a matching request are not transmitted from the real-time data processing terminal 3 to the centralized data processing server 7 via the near-terminal data processing device 5 and the distributed data processing terminal 6. This has the effect of preventing the user of the real-time data processing terminal 3 from being charged more communication fees than necessary.

〔supplement〕
When changing areas 2, 3, and 4, if the area is being changed so that its positional relationship with respect to the attention area "1" deviates from the positional relationship set on the area setting screen in Fig. 30, the display control unit 67 displays a warning to prevent erroneous changes. For example, when the areas are set as shown in Fig. 30 and the image of area 2 is changed, a semi-transparent red background color may be superimposed on the screen while the area is being changed to the right of the attention area.

In the above embodiment, in steps S217 and S218, only the distance between the latest object detection position and the past object detection position is used as the judgment material, but this is not limited to this. For example, the area difference between the latest object partial image and the past object partial image may also be used as the judgment material. Specifically, when the following two conditions (condition 1) and (condition 2) are satisfied, the process may proceed to step S219. According to this judgment, even if the current object detection position is close to the past object detection position, if there is an unnatural difference in the size of the detection area, it is possible not to recognize them as the same object (here, a person). This can further reduce transmissions (S115, S121) that should not require a matching request.
(Condition 1) The distance between the past object and the latest object is less than (or equal to) a predetermined threshold (same as step S218).
(Condition 2) The absolute value of the area difference between the past object and the latest object is less than (or equal to) a predetermined threshold value.
The area difference in condition 2 is the difference between the areas (width*height) calculated by the calculation unit 31 from the height and width of the partial images.

In the above embodiment, the standby state mark m1 is represented by an "!", but this is not limited to this. For example, it may be an hourglass icon or a comment such as "Verifying" or "Authenticating."

Although the captured image data has been described above, the present invention is not limited to this and may also be created image data created by the user without being captured. In this case, the captured image data and created image data are examples of "image data". Furthermore, the image acquisition terminal 2 is an example of a communication terminal, and image data may be created on the own terminal rather than acquired from an external source. Furthermore, the communication terminal may acquire sound data by collecting sound, may acquire temperature data by a temperature sensor, or may acquire humidity data by a humidity sensor.

The image data described above may be the captured image data itself, or partial image data representing a part of an object included in the captured image data (e.g., a person's face) or the entire object (e.g., a person's entire body). Furthermore, the image data may be the created image data itself, or partial image data representing a part of a character, such as a two-dimensional or three-dimensional character, included in the created image data, or the entire character.

Furthermore, each component such as the CPUs 301, 501, 701, etc. may be a single component or multiple components. Furthermore, there may be multiple image acquisition terminals 2, multiple distributed data processing terminals 6, and multiple centralized data processing servers 7. Furthermore, the distributed data processing terminal 6 may be a server.

Each function in the above-described embodiment can be realized by one or more processing circuits. Here, the "processing circuit" in the present embodiment can refer to a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, or an ASIC (Application Specific Integrated Circuit) designed to execute each of the above-described functions.
This includes devices such as digital signal processors (DSPs), field programmable gate arrays (FPGAs), systems on a chip (SOCs), GPUs, and conventional circuit modules.

2. Image acquisition terminal (an example of a communication terminal)
3 Real-time data processing terminal 5 Near-terminal data processing device 6 Distributed data processing terminal (an example of a setting device, an example of a display device)
7. Centralized data processing server (an example of a management device, an example of other devices)
31 Calculation unit 34 Image processing unit (an example of an image processing means)
35 Object detection unit (an example of an object detection means)
38 Connection unit (an example of an acquisition means)
51 Transmitting/receiving unit 61 Transmitting/receiving unit (an example of a receiving means, an example of a transmitting means)
62 Reception unit 65 Calculation unit (an example of a counting means)
67 Display control unit (an example of a display control means)
69 Storage/readout unit 71 Transmission/reception unit 74 Feature generation unit 75 Collation unit 100 Distributed processing system (an example of a processing system)
200 Intranet 517 Display (an example of a display means)
600 Internet 3001 Image sensor information management DB
3002 Cycle value management DB
3003 Image acquisition program DB
3004 Synthesis processing program management DB
3005 Distortion correction program management DB
3006 Service program management DB
3007 Object information management DB
3008 Object displacement management DB
6000 Storage unit 7001 Matching data management DB
8000 Storage unit 8001 Session management DB
8002 Authentication server management DB
9000 Storage unit 9001 Authentication management DB

JP 2006-113711 A

Claims (8)

a transmission means for transmitting image data representing a predetermined object to a management device that performs image matching;
a receiving means for receiving from the management device object identification information for identifying the object that is assigned as a result of the collation by the management device;
a display control means for simultaneously displaying a plurality of predetermined area images, which are images of predetermined areas among a plurality of areas included in a captured image obtained by capturing an image using a single imaging unit;
a receiving means for receiving a setting of each of a plurality of predetermined area images as a detectable range in which an object can be detected in the captured image;
having
the detectable range includes a collation range in which an image of an object can be collated;
the collation range is a predetermined area within the detectable range,
the transmitting means transmits, to the management device, first area identification information for identifying a first area among the plurality of areas and the object identification information of the object present in the first area in association with each other.
A processing system comprising: the size of the predetermined area of the collation range is set by a margin set inside the detectable range;
2. The processing system according to claim 1. The processing system according to claim 1 or 2, characterized in that, when the object moves from the first area to the second area, the transmission means transmits to the management device second area identification information for identifying the second area and the object identification information of the object present in the second area in association with each other. 4. The processing system according to claim 1,
The display control means causes the display means to display, for each moving direction, a number counted by a counting means that counts the number of movements of the object for each moving direction of the object at a predetermined time interval.
A processing system having The processing system according to any one of claims 1 to 4, characterized in that the image data is partial image data representing a portion of the object. The processing system according to any one of claims 1 to 5, characterized in that each of the multiple regions is a predetermined region among the multiple regions obtained from the omnidirectional image captured by the single imaging unit. A processing system according to any one of claims 1 to 5;
The management device;
A communication system having
The processing system includes:
a transmission means for transmitting image data representing a predetermined object to a management device that performs image matching;
a receiving means for receiving from the management device object identification information for identifying the object that is assigned as a result of the collation by the management device;
a display control means for simultaneously displaying a plurality of predetermined area images, which are images of predetermined areas among a plurality of areas included in a captured image obtained by capturing an image using a single imaging unit;
a receiving means for receiving a setting of each of a plurality of predetermined area images as a detectable range in which an object can be detected in the captured image;
having
the detectable range includes a collation range in which an image of an object can be collated;
the collation range is within a predetermined area of the detectable range,
A communication system characterized in that the transmission means associates first area identification information for identifying a first area among the multiple areas with the object identification information of the object present in the first area and transmits the associated information to the management device. A processing method executed by a processing system that performs a predetermined process on a management device that performs image matching, comprising:
a transmitting step of transmitting image data representing a predetermined object to the management device;
a receiving step of receiving, from the management device, object identification information for identifying the object that has been assigned as a result of the matching by the management device;
a display step of simultaneously displaying a plurality of predetermined area images, which are images of predetermined areas among a plurality of areas included in a captured image obtained by capturing an image using a single imaging unit;
a receiving step of receiving a setting of each of a plurality of predetermined area images as a detectable range in which an object can be detected in the captured image;
having
the detectable range includes a collation range in which an image of an object can be collated;
the collation range is a predetermined area within the detectable range,
A processing method characterized in that the transmission step includes a process of associating first area information for identifying a first area among the multiple areas with the object identification information of the object present in the first area and transmitting the associated information to the management device .