JP7706936B2 - Information processing device, control method for information processing device, and program - Google Patents

Description

The present invention relates to information processing technology based on event signals from an event-driven vision sensor.

In recent years, various imaging devices have been developed with the aim of reducing power consumption. Patent Document 1 discloses a technology that determines whether or not there is motion in multiple pixels based on the sum signal of multiple pixels, and controls whether or not to send the corresponding image data to subsequent processing. With this technology, an event in an image is determined from the presence or absence of motion in multiple pixels, and image data is not sent when no event is occurring, preventing subsequent processing from being performed, thereby reducing power consumption.

Event-driven vision sensors are also known in which pixels that detect changes in the intensity of incident light generate signals asynchronously. Event-driven vision sensors are advantageous in that they can operate at high speeds and with low power consumption compared to frame-type vision sensors that scan all pixels at a predetermined cycle, specifically image sensors such as CCDs and CMOSs. Technology related to such event-driven vision sensors is disclosed in Patent Document 2.

JP 2019-92022 A JP 2018-85725 A

While event-driven vision sensors have the advantages mentioned above, it is difficult to say that sufficient proposals have been made on peripheral technologies for reducing power consumption that take into account characteristics that differ from those of conventional vision sensors, such as frame-type vision sensors.

The present invention aims to provide a new technology for reducing power consumption using an event-driven vision sensor.

The information processing device of the present invention comprises an imaging means having an event-driven vision sensor that generates an event signal for each pixel in response to detection of a change in intensity of incident light, an information holding means capable of holding the event signal, and a control means for at least activating a signal processing means having a first operating mode in which the event signal can be processed and a second operating mode in which the signal processing means is in a power-saving state, wherein the information holding means is provided outside the imaging means , and the control means controls the signal processing means to transition to the first operating mode in response to the imaging means generating the event signal when the signal processing means is in the second operating mode, controls the signal processing means to transition to the second operating mode when the event signal is not input from the imaging means within a threshold time when the signal processing means is in the first operating mode, and controls the signal processing means to transition from the second operating mode to the first operating mode when the event signal is input from the imaging means, and causes the information holding means to hold the event signal input from the imaging means until the signal processing means transitions from the second operating mode to the first operating mode .

This invention makes it possible to provide a new technology for reducing power consumption using an event-driven vision sensor.

FIG. 1 is a diagram illustrating an example of the configuration of an information processing device according to a first embodiment. FIG. 4 is a sequence diagram showing a processing order of the information processing device according to the first embodiment. 4 is a flowchart showing a process of startup control in the first embodiment. FIG. 13 is a diagram showing an example of an image in which a time-series event signal is mapped. FIG. 11 is a diagram illustrating an example of the configuration of an information processing device according to a second embodiment. FIG. 13 is a diagram illustrating an example of an event mask region. 10 is a flowchart showing a process of startup control in the second embodiment. FIG. 13 is a diagram illustrating an example of the configuration of an information processing device according to a third embodiment. FIG. 13 is a sequence diagram showing a processing order of an information processing device according to a third embodiment. FIG. 13 is a diagram illustrating an example of the configuration of an information processing device according to a fourth embodiment. FIG. 13 is a sequence diagram showing a processing order of an information processing apparatus according to a fourth embodiment. FIG. 13 is a diagram illustrating an example of the configuration of an information processing device according to a fifth embodiment. 13 is a flowchart illustrating an event signal number measurement process according to the fifth embodiment.

An embodiment of the present invention will be described with reference to the drawings.
The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configurations of the embodiments may be appropriately modified or changed depending on the specifications of the device to which the present invention is applied and various conditions (conditions of use, environment of use, etc.). In addition, the present invention may be configured by appropriately combining parts of each of the embodiments described below. In the following embodiments, the same configurations are described with the same reference symbols.

First Embodiment
FIG. 1 is a block diagram showing an example of the configuration of an information processing apparatus 100 according to the first embodiment.
The image sensor 101 is an event-driven vision sensor including a sensor array 111 including sensors 111A, 111B, ... corresponding to pixels of an image, and a processing circuit 112 connected to the sensor array 111. The sensors 111A, 111B, ... include light receiving elements corresponding to pixels, and when a change in the intensity of light incident on the light receiving elements, i.e., a change in the luminance of a pixel is detected, a signal corresponding to the change in luminance is generated. The processing circuit 112 generates an event signal based on an output signal from the sensor that detects the change in luminance of the incident light. The event signal is generated as a signal including a time stamp, pixel identification information (the position of the sensor that detects the luminance change on the sensor array 111, i.e., pixel position information), and information indicating the polarity of the luminance change (increase or decrease in luminance value). At this time, the processing circuit 112 can also output the event signal including information indicating the luminance value of the pixel corresponding to the sensor of the sensor array 111.

When, for example, a subject moves within the field of view of the sensor array 111, the intensity of the light reflected or scattered by the subject changes. For this reason, it is possible to detect the movement of the subject in a time series manner, for example, by using an event signal generated based on a signal from a sensor corresponding to the edge of the subject. Also, when the conditions around the subject change within the field of view of the sensor array 111 (for example, a light is turned on in a dark room), the intensity of the light changes. For this reason, it is possible to detect changes in indoor conditions in a time series manner, for example, by using an event signal generated based on signals from each sensor corresponding to the image of the entire subject.

Furthermore, an event-driven vision sensor such as the image sensor 101 can operate at low power and high speed compared to a frame-type vision sensor (e.g., a CCD or CMOS image sensor) that scans all pixels at a predetermined cycle. This is because only those sensors 111A, 111B, ... that make up the sensor array 111 that detect a luminance change generate a signal of that luminance change. Since no signal is generated by sensors that do not detect a luminance change, the processing circuit 112 can process at high speed only the signals of sensors that detect a luminance change and output an event signal.

When the start-up control unit 102 receives an event signal output from the image sensor 101, the start-up control unit 102 stores (records) the event signal in an information storage unit 103 (described later) and outputs a start-up signal to an event signal processing unit 104 (described later). The operation of the start-up control unit 102 will be described in detail later. Note that the functions of the start-up control unit 102 may be realized by a CPU or the like executing a program according to this embodiment.
The information storage unit 103 is an information recording unit having a capacity sufficient to store a plurality of event signals. The operation of the information storage unit 103 will be described in detail later.

The event signal processing unit 104 receives a start-up signal when the start-up signal is transmitted from the start-up control unit 102, and also receives an event signal when the event signal is transmitted from the image sensor 101 via the start-up control unit 102. In this embodiment, the event signal processing unit 104 has at least two operation modes, a first operation mode and a second operation mode. The first operation mode is an operation mode in which the received event signal can be processed (hereinafter referred to as a normal processing mode). The second operation mode is an operation mode in which the power saving state is entered and signal processing is stopped (hereinafter referred to as a power saving mode). The event signal processing unit 104 also has a function of notifying the start-up control unit 102 of a state notification signal indicating whether the current operation mode is the power saving mode or the normal processing mode. The operation of the event signal processing unit 104 will be described in detail later.

Next, the flow of operations of the information processing device 100 in the first embodiment will be described in detail with reference to the sequence diagram in FIG. 2. The sequence diagram in FIG. 2 shows a flow in which the event signal processing unit 104 starts in a power saving mode state, transitions to a normal processing mode in response to an event detection by the image sensor 101, and then transitions back to the power saving mode. Note that in the sequence diagrams and flowcharts shown hereafter, the processing steps are represented by the letter S.

In the imaging sensor 101, when the sensor detects a luminance change within the angle of view of the sensor array 111, an event signal corresponding to a pixel of the sensor (referred to as a target pixel) is generated by the processing circuit 112. Then, in S201, the imaging sensor 101 outputs the event signal of the target pixel to the start-up control unit 102. Note that in this embodiment, an example is given in which the generated event signal is output as is, but this is not limited to this. For example, the processing circuit 112 of the imaging sensor 101 may output the generated event signal after reducing the amount of data by compressing it using a known data compression method.

After receiving the event signal output by the image sensor 101, the startup control unit 102 outputs a startup signal to the event signal processing unit 104, which is in the power saving mode, in order to transition to the normal processing mode in S202.

The flow of the start signal transmission process will be described in detail below with reference to the flowchart of FIG.
After receiving the event signal, the start control unit 102 checks the current operation mode of the event signal processing unit 104 by referring to the state notification signal in S301. If the event signal processing unit 104 is in the power saving mode, the start control unit 102 proceeds to S302, whereas if the event signal processing unit 104 is in the normal processing mode, the start control unit 102 ends the process of the flowchart in Fig. 3. The state notification signal is a signal that can be expressed by a small amount of data of 1 bit to several bits, and is a signal notified by the event signal processing unit 104 when the event signal processing unit 104 changes its operation mode. The start control unit 102 is capable of internally holding the state notification signal sent from the event signal processing unit 104, and can check the current operation mode of the event signal processing unit 104 by referring to the held state notification signal.

When the process proceeds to S302 because the current operating mode of the event signal processing unit 104 is the power saving mode, the start-up control unit 102 determines whether or not a start-up signal for transitioning the event signal processing unit 104 to the normal processing mode has not yet been transmitted. If the start-up signal has not yet been transmitted, the start-up control unit 102 proceeds to S303, and if the start-up signal has already been transmitted, the start-up control unit 102 ends the process of the flowchart in FIG. 3. For example, the start-up control unit 102 may hold flag information indicating whether or not a start-up signal has been transmitted, and determine whether or not a start-up signal has not yet been transmitted by referring to the flag information.

When the process proceeds to S303, the start-up control unit 102 generates a start-up signal and transmits it to the event signal processing unit 104. The start-up signal can be expressed with a small amount of data, from one bit to a few bits. Upon receiving the start-up signal, the event signal processing unit 104 transitions from the power saving mode to the normal processing mode, and upon completing the transition to the normal processing mode, transmits to the start-up control unit 102 a status notification signal indicating that the current operating mode is the normal processing mode.

Returning to the sequence diagram of FIG.
If an event signal is input when the current operation mode of the event signal processing unit 104 is the power saving mode, in S203, the start-up control unit 102 performs a write process to cause the event signal to be stored in the information storage unit 103. At this time, the start-up control unit 102 executes in parallel the transmission of the start-up signal to the event signal processing unit 104 and the write process of the event signal to the information storage unit 103. Note that, if the current operation mode of the event signal processing unit 104 is the normal processing mode, the start-up control unit 102 does not perform the write process to cause the input event signal to be stored in the information storage unit 103, but transmits the event signal directly to the event signal processing unit 104.

After receiving the startup signal transmitted by the startup control unit 102, the event signal processing unit 104 transitions to a normal processing mode in which the event signal can be processed. Then, when the transition to the normal processing mode is completed, in S204, the event signal processing unit 104 transmits a state notification signal to the startup control unit 102 to notify that the transition from the power saving mode to the normal processing mode has been completed.

When the start control unit 102 receives the state notification signal from the event signal processing unit 104, it recognizes that the event signal processing unit 104 has transitioned to the normal processing mode. Then, the start control unit 102 starts a process of reading the event signals stored in the information storage unit 103 in FIFO order (first-in first-out order). Furthermore, in S205, the start control unit 102 transmits the event signals read from the information storage unit 103 to the event signal processing unit 104. Note that, when a new event signal is input from the image sensor 101 while an event signal remains in the information storage unit 103, the start control unit 102 performs a write process to hold the new event signal while reading the event signal remaining in the information storage unit 103. Also, when a new event signal is input while no event signal remains in the information storage unit 103, the start control unit 102 does not perform a write process to the information storage unit 103, but transmits the new event signal directly to the event signal processing unit 104.

After receiving the event signal transmitted from the start-up control unit 102, the event signal processing unit 104 executes processing based on the event signal in S206. In this embodiment, the event signal processing unit 104 executes processing using position information of a pixel whose luminance value has changed, or the luminance value of the pixel, or a timestamp, contained in the event signal.

Now, an example of event signal processing executed in the event signal processing unit 104 will be described with reference to FIG. 4. FIG. 4 shows an example of generating an image that maps the change in event signal over time when an object (subject) moves within the field of view of the sensor array 111 by using position information of pixels whose luminance values have changed, the luminance values of the pixels, and a timestamp.

Figure 4(a) is a diagram showing pixel arrangement 1040 corresponding to the sensor arrangement on sensor array 111, and the position of each pixel whose luminance value has changed at time t0 is shown as pixel position 1140. That is, Figure 4(a) is a diagram showing the position information of each pixel whose luminance value has changed at time t0 due to, for example, the appearance of an object in the angle of view of sensor array 111, as pixel position 1140 on pixel arrangement 1040 of sensor array 111. Note that the event signal at time t0 includes a timestamp indicating time t0, pixel identification information (pixel position information) indicating pixel position 1140, and information indicating the polarity of the luminance change, and these are input to event signal processing unit 104.

FIG. 4(b) is a diagram showing pixel arrangement 1040 corresponding to the sensor arrangement on the sensor array 111, similar to FIG. 4(a). FIG. 4(b) shows pixel position 1140 of each pixel whose luminance value has changed at time t0, and pixel position 1241 of each pixel whose luminance value has changed at time t1 after time t0. That is, FIG. 4(b) shows that the position of each pixel whose luminance value has changed has moved from pixel position 1140 at time t0 to pixel position 1241 at time t1 due to the movement of an object within the angle of view of the sensor array 111. Note that the event signal at time t1 includes a timestamp representing the time t1, pixel identification information (pixel position information) representing the pixel position 1241, and information indicating the polarity of the luminance change, and these are input to the event signal processing unit 104.

FIG. 4(c) is a diagram showing pixel arrangement 1040 corresponding to the sensor arrangement on the sensor array 111, similar to FIG. 4(a). FIG. 4(c) shows pixel position 1241 of each pixel whose luminance value has changed at time t1, and pixel position 1342 of each pixel whose luminance value has changed at time t2, which is further after time t1. That is, FIG. 4(c) shows that the position of each pixel whose luminance value has changed has moved from pixel position 1241 at time t1 to pixel position 1342 at time t2 due to further movement of an object within the angle of view of the sensor array 111. Note that the event signal at time t2 includes a timestamp representing the time t2, pixel identification information (pixel position information) representing the pixel position 1342, and information indicating the polarity of the luminance change, and these are input to the event signal processing unit 104.

Figure 4(d) is a diagram showing an example of image 1043 generated by the event signal processing unit 104 by mapping the time change of the event signal in correspondence with the pixel arrangement of the sensor array 111 based on the position information of the pixel whose luminance value has changed, the luminance value of the pixel, and the timestamp. That is, image 1043 in Figure 4(d) is an example of an image generated by mapping pixels whose luminance value has changed between times t0 and t2 using event signals detected and input at consecutive times t0 to t2 as shown in Figures 4(a) to 4(c).

In the case of the mapping image example shown in FIG. 4(d), the event signal processing unit 104 first generates an image in which a pixel at a position corresponding to pixel position 1140 in FIG. 4(a), that is, a pixel in which a luminance change has occurred, is mapped based on the pixel position information included in the event signal at time t0. Next, the event signal processing unit 104 generates an image in which a pixel corresponding to pixel position 1241 in FIG. 4(b) is mapped based on the pixel position information included in the event signal at time t1, that is, a pixel position in which a change has occurred from the pixel position information of pixel position 1241 at time t1. Next, the event signal processing unit 104 generates an image in which a pixel corresponding to pixel position 1342 in FIG. 4(c) is mapped based on the pixel position information included in the event signal at time t2, that is, a pixel position in which a change has occurred from the pixel position information of pixel position 1241 at time t1. This generates an image 1043 in which pixel 1440 corresponding to pixel position 1140, pixel 1441 corresponding to pixel position 1241, and pixel 1442 corresponding to pixel position 1342 are mapped, as shown in FIG. 4(d).

That is, when generating the mapping ping image of FIG. 4(d), the event signal processing unit 104 acquires pixel position information, pixel brightness value, and timestamp from each event signal at time t0 to t2. Then, if the acquired pixel position information is the same as the pixel position information acquired at the previous time (for example, when the current time is t1, the previous time is t0), the event signal processing unit 104 records the information with the older timestamp. Note that at this time, the medium for recording the pixel position information and timestamp may be a memory provided in the event signal processing unit 104. This allows the event signal processing unit 104 to generate an image in which pixels with a change in brightness value are mapped by reading the recorded pixel position information and timestamp.

Returning to the sequence diagram of FIG. 2, the flow in which the event signal processing unit 104 transitions from the normal processing mode to the power saving mode will be described.
The start-up control unit 102 compares the time tn from when the n-th event signal output from the image sensor 101 is input until the next event signal is input from the image sensor 101 with a certain time (threshold time _Tth ) defined as a threshold for time. Here, if the time tn exceeds the threshold time _Tth , that is, if the next event signal is not input from the image sensor 101 even when the time tn reaches the threshold time _Tth , the start-up control unit 102 determines that an event has not occurred within the angle of view of the sensor array 111 for a certain time. Then, if the start-up control unit 102 determines that an event has not occurred for a certain time, in S207, the start-up control unit 102 transmits a stop signal to the event signal processing unit 104 to cause the normal processing mode to transition to the power saving mode. On the other hand, if an event signal is output from the image sensor 101 within the threshold time (i.e., within the certain time), the start-up control unit 102 does not transmit a stop signal to cause the normal processing mode to transition to the power saving mode. The threshold time _Tth for determining whether or not to output a stop signal may be set in advance, or may be changed by an instruction from an external device or the like. Also, the stop signal output by start-up control unit 102 is assumed to be prepared in advance as a signal independent of the start-up signal.

When the event signal processing unit 104 receives a stop signal from the start-up control unit 102, it transitions the operation mode from the normal processing mode to the power saving mode. Then, after transitioning to the power saving mode, in S208, the event signal processing unit 104 notifies the start-up control unit 102 of a state notification signal indicating that the current operation mode is the power saving mode.

In this way, the startup control unit 102 can instantly control the event signal processing unit 104 to a normal processing mode or a power saving mode depending on whether an event signal generated asynchronously in the image sensor 101, which is an event-driven vision sensor, is input.

In addition, according to the information processing device 100 of this embodiment, the operating mode of the event signal processing unit 104 is controlled using an event signal from a low-power event-driven vision sensor as a startup trigger, thereby making it possible to further reduce power consumption as an information processing device.
In the present embodiment, an example of application to the information processing device 100 has been given, but the present invention is not limited to this and can also be applied to, for example, a surveillance camera.

Second Embodiment
Next, an information processing apparatus according to a second embodiment will be described.
In the first embodiment, an example was given in which the start control unit 102 controls the operation mode of the event signal processing unit 104 in response to the generation of an event signal by an event-driven vision sensor, depending on the current operation mode. In contrast, in the second embodiment, a partial area on the sensor array 111 is defined as an event mask area, and an event signal corresponding to the inside of the event mask area is not processed as a normal event signal.
The operation of the information processing apparatus of the second embodiment will be described with reference to Fig. 5 to Fig. 7. Note that the description of the same configuration and processing as in the first embodiment will be omitted.

FIG. 5 is a block diagram showing an example of the configuration of an information processing device 500 according to the second embodiment.
The mask area setting unit 501 sets an event mask area for the start control unit 502. The event mask area is expressed by pixel position information on the sensor array 111, and can be set, for example, as a rectangular area or an area of any shape, and the size of the area, the number of areas, and the positions of the areas can also be set arbitrarily. The event mask area can also be set from outside the mask area setting unit 501. Although details will be described later, the event mask area is an area in which the start control unit 502 processes the event signal output from the image sensor 101 as an invalid event signal.

In the second embodiment, when the start-up control unit 502 receives an event signal from the image sensor 101, it determines whether the event signal is outside the event mask area set by the mask area setting unit 501. If the event signal is outside the event mask area, the start-up control unit 502 causes the information holding unit 103 to hold the event signal, and transmits a start-up signal if the event signal processing unit 104 is in power saving mode. On the other hand, if the event signal is within the event mask area, the start-up control unit 502 does not hold the event signal in the information holding unit 103 as an invalid event signal, and does not transmit a start-up signal to the event signal processing unit 104.

Figure 6 shows pixel arrangement 601 corresponding to each sensor on sensor array 111. Figure 6 also shows event mask area 610 set by mask area setting unit 501 and pixel positions 611 of each pixel whose luminance value has changed on pixel arrangement 601 of sensor array 111. In the example of Figure 6, a rectangular area of 5 x 5 pixels represented by coordinates (X1, Y1) to (X5, Y5) is set as event mask area 610. Furthermore, pixel positions 611 show each pixel whose luminance value has changed at time tm.

As described above, the event mask area is an area in which the start control unit 502 invalidates event signals corresponding to pixel positions within the event mask area, and treats event signals at pixel positions outside the event mask area as valid signals. Therefore, when the start control unit 502 receives an event signal from the image sensor 101, it acquires position information of the pixel included in the event signal and determines whether the event signal is included in the event mask area 610. Then, the start control unit 502 determines whether to output a start signal to the event signal processing unit 104 according to the determination result. For example, if the event signal is not included in the event mask area 610, the start control unit 502 outputs a start signal to the event signal processing unit 104 as in the first embodiment described above. On the other hand, if the event signal is included in the event mask area 610, the start control unit 502 does not output a start signal to the event signal processing unit 104.

FIG. 7 is a flowchart showing a detailed process flow of the start-up control unit 502 in the second embodiment.
In S701, when the start control unit 502 receives an event signal from the image sensor 101, the start control unit 502 acquires pixel position information (coordinates (Xm, Ym)) contained in the event signal.
Next, in S702, the start control unit 502 compares the pixel position information acquired in S701 with the set event mask area, and judges whether the coordinates (Xm, Ym) of the pixel position information are outside the coordinates (X1, Y1) to (X5, Y5) of the event mask area. If the pixel position information is outside the event mask area, the start control unit 502 advances the process to S301, and thereafter performs the same processes as S301 to S303 in FIG. 3.

On the other hand, if the acquired pixel position information is not outside the event mask area, that is, if it is within the event mask area, the start-up control unit 502 proceeds to S703. Then, in S703, the start-up control unit 502 discards the event signal determined to be within the event mask area in S702 without storing it in the information storage unit 103, and ends the processing of the flowchart in FIG. 7.

As described above, the information processing device 500 of the second embodiment sets an event mask area, and determines whether the event signal output from the image sensor 101 is valid or invalid depending on whether the pixel position of the event signal is within the event mask area. According to the second embodiment, the event signal processing unit 104 can be controlled to a normal processing mode or a power saving mode depending on whether the event signal is within the event mask area. Then, if the event signal is within the event mask area, the event signal processing unit 104 does not transition to the normal processing mode, thereby reducing power consumption.

The information processing device 500 of this embodiment may be applied, for example, in a use case where the information processing device 500 is mounted on a fixed-point monitoring camera, in which when the area to be monitored (area to be viewed) is limited, the remaining area is set as an event mask area.
Since substantially no event detection is performed within the event mask region, it is possible to suppress the number and frequency of event detections. In addition, the suppression of the number and frequency of event detections reduces event signal processing and the number of state transitions to the normal processing mode of the event signal processing unit 104, which is expected to have a power saving effect on the event signal processing unit 104.

In the second embodiment, an example was described in which an event signal in the event mask area is not processed as an invalid event signal. However, it is also possible to handle only event signals in the event mask area as valid event signals. In addition, for example, a different process may be performed on an event signal in the event mask area than on an event signal outside the event mask area. For example, an event signal generated in the event mask area may be subjected to signal processing with a lighter load than that performed on an event signal generated outside the event mask area. In this case, for example, when sending an event signal to the event signal processing unit 104, the start control unit 502 notifies the event signal processing unit 104 as to whether the event signal is a signal inside or outside the event mask area. Then, the event signal processing unit 104 may be able to reduce power consumption by performing a light-load process on an event signal in the event mask area compared to a case where a heavy-load signal processing is performed.

Third Embodiment
Next, an information processing apparatus according to a third embodiment will be described with reference to Fig. 8 and Fig. 9. Note that descriptions of the same configuration and processing as those of the first and second embodiments will be omitted.
FIG. 8 is a block diagram showing an example of the configuration of an information processing apparatus 800 according to the third embodiment.
The image sensor 801 is an event-driven vision sensor including the sensor array 111 as described above, and a processing circuit 812 connected to the sensor array 111 .

In addition to the function of generating an event signal as described above, the processing circuit 812 of the third embodiment has a function of outputting an event notification signal to notify the start control unit 802 that the event signal has been generated prior to outputting the event signal. The processing circuit 812 also has a function of inputting and outputting the event notification signal and the event signal via independent signal lines. The processing circuit 812 outputs the event signal to the start control unit 802 only when the start control unit 802 is in a state where it can receive the event signal. The event notification signal can be expressed with a small amount of data, such as one bit or a few bits. The path of the event notification signal from the image sensor 801 to the start control unit 802 may use a data transmission bus for the event signal, or may use a bus for transmitting the event notification signal that is provided separately from the data transmission bus.

When the start-up control unit 802 receives an event notification signal from the imaging sensor 801, it transmits a start-up signal to the event signal processing unit 104. At the same time, the start-up control unit 802 checks the current operating mode of the event signal processing unit 104 using the state notification signal, and notifies the processing circuit 812 whether or not the event signal processing unit 104 is in a state in which it can receive an event signal. The operation of the imaging sensor 801 and the start-up control unit 802 in the third embodiment will be described in detail later.

The flow of operations of the information processing device 800 in the third embodiment will be described in detail with reference to the sequence diagram in FIG. 9. The sequence diagram in FIG. 9 shows a flow in which the event signal processing unit 104 starts in a power saving mode state, transitions to a normal processing mode in response to an event detection by the image sensor 801, and then transitions back to the power saving mode.

In the image sensor 801 of the third embodiment, when the sensor detects a luminance change within the angle of view of the sensor array 111, the processing circuit 812 generates an event notification signal and outputs it to the start-up control unit 802 in S901.
When the start-up control unit 802 receives the event notification signal from the image sensor 801, it checks the current operation mode of the event signal processing unit 104 based on the state notification signal. Since the event signal processing unit 104 is in the power saving mode at this time, the start-up control unit 802 transmits a start-up signal to the event signal processing unit 104 in S902 to cause the event signal processing unit 104 to transition to the normal processing mode.

When the event signal processing unit 104 receives a startup signal from the startup control unit 802, it transitions to a normal processing mode in which it can process event signals. Then, when the transition to the normal processing mode is complete, in S903, the event signal processing unit 104 transmits a state notification signal to the startup control unit 802 to notify that the transition from the power saving mode to the normal processing mode has been completed.

When the start-up control unit 802 receives a state notification signal indicating that the normal processing mode is selected from the event signal processing unit 104, in S904, the start-up control unit 802 notifies the processing circuit 812 of the image sensor 801 that an event signal can be received. After receiving notification from the start-up control unit 802 that an event signal can be received, the processing circuit 812 detects a change in pixel luminance within the angle of view of the sensor array 111, and in S905 generates an event signal in response to the detected change in luminance and outputs it to the start-up control unit 802. That is, in the case of the third embodiment, after outputting the event notification signal, the processing circuit 812 of the image sensor 801 does not output the change in pixel luminance that occurs during that time as an event signal until it receives notification from the start-up control unit 802 that an event signal can be received.

When the start control unit 802 receives the event signal output from the image sensor 801, in S906, it transmits the event signal directly to the event signal processing unit 104. The subsequent processes of S206, S207, and S208 are the same as those described above.
Thereafter, when the start-up control unit 802 receives a state notification signal indicating that the event signal processing unit 104 has transitioned to a power saving mode, in S907 it notifies the image sensor 801 that it is in a state in which it cannot receive an event signal.

Incidentally, although the information storage unit 103 is not shown in the configuration shown in FIG. 8, the information processing device 800 of the third embodiment may also have the information storage unit 103 similar to that described above. In this case, the start control unit 802 may cause the information storage unit 103 to store the event signal generated after receiving the event notification signal from the image sensor 801 and until the event signal processing unit 104 is able to process the event signal. However, in the case of the information processing device 800 of the third embodiment, if the event signal generated before the event signal processing unit 104 is able to process the event signal is not transmitted to the event signal processing unit 104, the information storage unit 103 may not necessarily be provided.

As described above, in the information processing device 800 of the third embodiment, a dedicated signal is used to notify the occurrence of an event, and the event signal processing unit 104 transmits an event signal from the image sensor 801 only when in normal processing mode. This makes it possible to reduce power consumption even in the information processing device 800 of the third embodiment.

In the third embodiment, it is also possible to further set an event mask area for determining whether an event signal is valid or invalid, as in the second embodiment. That is, in the third embodiment, too, power consumption can be further reduced by controlling the event signal processing unit 104 to a normal processing mode or a power saving mode depending on whether the event signal is within the event mask area.

Fourth Embodiment
Next, an information processing apparatus according to a fourth embodiment will be described with reference to Fig. 10 and Fig. 11. Note that descriptions of the same configurations and processes as those of the above-mentioned embodiments will be omitted as appropriate.
FIG. 10 is a block diagram showing an example of the configuration of an information processing apparatus 1000 according to the fourth embodiment.
The image sensor 1001 is an event-driven vision sensor including the sensor array 111 and the processing circuit 1012 connected to the sensor array 111. In the fourth embodiment, the processing circuit 1012 has a function of generating an event signal as described above, and also a function of outputting an event notification signal for notifying the start control unit 802 that the event signal has been generated. In the fourth embodiment, the image sensor 1001 also includes an information holding unit 1013 for holding the event signal. In the configuration example of FIG. 10, for example, the processing circuit 1012 includes the information holding unit 1013, but the processing circuit 1012 and the information holding unit 1013 may be connected as separate configurations within the image sensor 1001. In addition, the structure of the image sensor 1001 according to the fourth embodiment may be a stacked type in which the sensor array 111, the processing circuit 1012, and the information holding unit 1013 are manufactured separately and stacked.

In the fourth embodiment, the processing circuit 1012 notifies the start-up control unit 1002 that an event signal has been generated, and then writes the event signals generated during the period until the event signal processing unit 104 transitions to the normal processing mode to the information storage unit 1013. The operations of the image sensor 1001 and the start-up control unit 1002 will be described in detail later.

Next, the flow of operations of the information processing device 1000 in the fourth embodiment will be described in detail with reference to the sequence diagram of FIG. 11. The sequence diagram of FIG. 11 shows a flow in which the event signal processing unit 104 starts in a power saving mode state, transitions to a normal processing mode in response to an event detection by the image sensor 1001, and then transitions back to the power saving mode.

In the fourth embodiment of the image sensor 1001, when the sensor detects a luminance change within the angle of view of the sensor array 111, the processing circuit 1012 generates an event notification signal in S1101 and outputs it to the startup control unit 1002. When the startup control unit 1002 receives the event notification signal, it checks the current operation mode of the event signal processing unit 104 from the state notification signal. Since the event signal processing unit 104 is in the power saving mode at this time, the startup control unit 1002 transmits a startup signal to the event signal processing unit 104 in S1102 to transition to the normal processing mode. The processing circuit 1012 of the image sensor 1001 at this time also performs processing to write the event signal of the target pixel in the sensor array 111 where the luminance change has been detected to the information holding unit 1013 in S1103.

When the start-up control unit 1002 receives a status notification signal indicating a transition to the normal processing mode from the event signal processing unit 104 in S204, it sends the status notification signal to the processing circuit 1012 of the image sensor 1001 in S1104. By receiving the status notification signal from the start-up control unit 1002, the processing circuit 1012 of the image sensor 1001 recognizes that the event signal processing unit 104 has transitioned to the normal processing mode.

When the processing circuit 1012 of the image sensor 1001 recognizes that the event signal processing unit 104 has transitioned to the normal processing mode, it starts the process of reading out the event signal stored in the information storage unit 1013 in FIFO order. Then, the event signal read out from the information storage unit 1013 is sent to the start-up control unit 1002 in S1105. When the start-up control unit 1002 receives an event signal from the image sensor 1001, it transmits the event signal to the event signal processing unit 104 in S1106. Note that when a new luminance change signal is input from the sensor array 111 while an event signal remains in the information storage unit 1013, the processing circuit 1012 reads out the event signal remaining in the information storage unit 1013 and writes an event signal corresponding to the new luminance change. Furthermore, if a new luminance change signal is input when no event signal remains in the information storage unit 1013, the processing circuit 1012 sends the new event signal corresponding to the luminance change directly to the start control unit 1002 without writing it to the information storage unit 1013. The subsequent processing of S206, S207, and S208 is the same as described above.

As described above, the information processing device 1000 of the fourth embodiment includes an information storage unit 1013 in the image sensor 1001, and transmits an event signal from the image sensor 1001 only when the event signal processing unit 104 is in normal processing mode. This makes it possible for the information processing device 1000 of this embodiment to reduce power consumption.

In the fourth embodiment, as in the second embodiment, it is also possible to set an event mask area for determining whether an event signal is valid or invalid. That is, in the fourth embodiment, too, the event signal processing unit 104 can be controlled to a normal processing mode or a power saving mode depending on whether the event signal is within the event mask area, thereby reducing power consumption.

Fifth embodiment
Next, as a fifth embodiment, an example will be described in which the event signal processing unit 104 is transitioned from a power saving mode to a normal processing mode when the number of event signals generated within a certain period of time exceeds a number of event signals predefined as a threshold value.
Hereinafter, an information processing apparatus according to the fifth embodiment will be described with reference to Fig. 12 and Fig. 13. Note that descriptions of the same configurations and processes as those of the above-mentioned embodiments will be omitted as appropriate.
FIG. 12 is a block diagram showing an example of the configuration of an information processing apparatus 1200 according to the fifth embodiment.

In the fifth embodiment, the start-up control unit 1202 generates an event reception notification signal indicating that an event signal has been received, and sends the event reception notification signal to the event number measurement unit 1201, each time an event signal is input from the image sensor 101. In other words, the event reception notification signal is a notification signal that is generated by the start-up control unit 1202 each time an event signal is received from the image sensor 101.

The event number measurement unit 1201 has an event number counter that counts the event reception notification signals sent from the start control unit 1202 to acquire the number of event signals generated within a predefined specified time. Hereinafter, the specified time is assumed to be Δt. When the number of event signals measured within the specified time Δt exceeds a number (threshold number _Nth ) predefined as a threshold, the event number measurement unit 1201 generates a notification signal indicating this (event number reached notification signal) and sends it to the start control unit 1202. Note that the measurement of the specified time Δt can be calculated, for example, by counting the number of clocks, but is not limited to this.

When start control section 1202 receives the event number arrival notification signal from event number measurement section 1201, it transmits a start signal for transitioning event signal processing section 104 to the normal processing mode.
Although FIG. 12 shows an example in which the event number measuring section 1201 and the start control section 1202 are configured separately, the start control section 1202 may have the function of the event number measuring section 1201 .

FIG. 13 is a flowchart showing a detailed process flow in the event number measurement unit 1201 of the fifth embodiment.
In S1301, the event number measurement unit 1201 waits until an event reception notification signal is input from the start control unit 1202. Then, upon receiving the event reception notification signal, the event number measurement unit 1201 advances the process to S1302.
In S1302, the event count measurement unit 1201 determines whether the event reception notification signal occurred within a specified time Δt defined for measuring the number of event signals. If the event count measurement unit 1201 determines that the event reception notification signal occurred within the specified time Δt, the process proceeds to S1303. On the other hand, if the event reception notification signal occurs beyond the specified time Δt, the process proceeds to S1306.

In S1303, the event number measurement unit 1201 counts up the number of input event reception notification signals, that is, the counter value that counts the number of event signals.
Next, in S1304, the event number measurement unit 1201 determines whether the count value of the event reception notification signal, i.e., the count value of the number of event signals, is greater than the threshold number _Nth . If the event number measurement unit 1201 determines that the count value of the number of event signals is greater than the threshold number _Nth , the process proceeds to S1305, whereas if the count value of the number of event signals is equal to or less than the threshold number (equal to or less than the threshold number _Nth ), the process returns to S1301.

In S1305, the event number measurement unit 1201 determines that an event signal exceeding the threshold number N _th has occurred within the specified time Δt, and sends an event number arrival notification signal to the start control unit 1202.
Thereafter, the process proceeds to S1306, where the event number measurement unit 1201 resets the specified time Δt and the number of event signals, and then ends the process of the flowchart in FIG.

The specified time Δt and the threshold number N _th for counting the number of event signals can be set to any value. These arbitrary values can also be set to the event number measurement unit 1201 from outside.
When the event signal processing unit 104 is in the power saving mode, the start-up control unit 1202, which has received the event number arrival notification signal, transmits a start-up signal for transitioning to the normal processing mode when the event signal processing unit 104 is in the power saving mode. On the other hand, when the event signal processing unit 104 is in the normal processing mode, the start-up control unit 1202 ignores the event number arrival signal input from the event number measurement unit 1201.

As described above, in the information processing device 1200 of the fifth embodiment, the event signal processing unit 104 can be controlled to a normal processing mode or a power saving mode depending on the number of event signals generated asynchronously within a specified time by the image sensor 101. According to the fifth embodiment, when the event signal processing unit 104 is in the power saving mode, if the number of event signals is equal to or less than a threshold number, the event signal processing unit 104 does not switch to the normal processing mode, which makes it possible to further reduce power consumption.

The functions described in the fifth embodiment can also be applied to each of the first to fourth embodiments described above. As an example, when the functions of the fifth embodiment are applied to the second embodiment, it is possible to control the event signal processing unit 104 to the normal processing mode when the number of event signals reaches a threshold number within the event mask area. Also, for example, if the event mask area is handled in the same way as in the second embodiment, and the event signal processing unit 104 is controlled to the normal processing mode when the number of event signals outside the event mask area reaches a threshold number, it is possible to further reduce power consumption.

The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.
The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main characteristics.

100: Information processing device, 101: Image sensor, 102: Startup control unit, 103: Information storage unit, 104: Event signal processing unit, 111: Sensor array, 112: Processing circuit

Claims (13)

an image capturing means having an event-driven vision sensor that generates an event signal for each pixel in response to detection of a change in the intensity of incident light;
an information storage means capable of storing an event signal;
a control means for at least activating a signal processing means having a first operation mode capable of processing the event signal and a second operation mode in which the device is in a power saving state;
the information storage means is provided outside the imaging means,
The control means
When the signal processing means is in the second operation mode, the signal processing means is controlled to transition to the first operation mode in response to the image capturing means generating the event signal;
when the event signal is not input from the imaging means within a threshold time, the signal processing means is controlled to transition to the second operation mode ;
When the event signal is input from the imaging means, the signal processing means is controlled to transition from the second operation mode to the first operation mode, and the event signal input from the imaging means is held in the information holding means until the signal processing means transitions from the second operation mode to the first operation mode.
23. An information processing apparatus comprising: The information processing device according to claim 1, further comprising the signal processing means. 2. The information processing apparatus according to claim 1, wherein the control means transmits the event signal read from the information holding means to the signal processing means after the signal processing means transitions from the second operation mode to the first operation mode. The information storage means is provided in the imaging means,
3. The information processing device according to claim 1, wherein the imaging means causes the event signal to be held in the information holding means until the control means controls the signal processing means to transition from the second operation mode to the first operation mode. when the image capturing means generates the event signal, the image capturing means outputs a notification indicating that the event signal has been generated to the control means;
the control means controls the signal processing means to transition to the first operation mode when the notification that the event signal has been generated is input from the imaging means when the signal processing means is in the second operation mode, and outputs a notification indicating that the signal processing means has transitioned to the first operation mode after transitioning from the second operation mode to the first operation mode to the imaging means;
5. The information processing device according to claim 4, wherein the imaging means retains the event signal in the information retaining means until the notification indicating that the signal processing means has transitioned to the first operation mode is input from the control means, and transmits the event signal retained by the information retaining means to the signal processing means via the control means after the notification indicating that the signal processing means has transitioned to the first operation mode is input from the control means . The control means
A mask area is stored to determine whether the event signal is valid or invalid.
An information processing device according to any one of claims 1 to 5, characterized in that, when the signal processing means is in the second operation mode, the signal processing means determines whether the event signal input from the imaging means is valid or invalid based on the mask area, and when the event signal is valid, the signal processing means is controlled to transition to the first operation mode. a region setting means for setting the mask region represented by a plurality of pixel positions corresponding to the positions of a plurality of sensors of the event-driven vision sensor of the imaging means,
7. The information processing apparatus according to claim 6 , wherein said control means holds information about the mask area set by said area setting means. when the image capturing means generates the event signal, the image capturing means outputs a notification indicating that the event signal has been generated to the control means prior to outputting the event signal;
3. The information processing device according to claim 1, wherein the control means controls the signal processing means to transition to the first operation mode when the notification indicating that the event signal has been generated is input from the imaging means when the signal processing means is in the second operation mode. 9. The information processing apparatus according to claim 8, wherein the image capturing means transmits the event signal and a notification indicating that the event signal has been generated to the control means using different signal lines. 10. The information processing device according to claim 1, wherein when the signal processing means is in the second operating mode, when the number of the event signals generated by the imaging means within a specified time exceeds a threshold value, the control means controls the signal processing means to transition to the first operating mode. The information processing device according to claim 10, further comprising a measuring means for measuring the number of the event signals generated by the imaging means, and notifying the control means that the number of the event signals has exceeded the threshold when the number of the measured event signals within the specified time exceeds the threshold . A method for controlling an information processing device, comprising:
The information processing device includes an imaging means having an event-driven vision sensor that generates an event signal for each pixel in response to detection of a change in intensity of incident light, an information holding means capable of holding the event signal, and a control means that at least starts up a signal processing means having a first operation mode capable of processing the event signal and a second operation mode in a power saving state;
the information storage means is provided outside the imaging means,
a step of controlling, when the signal processing means is in the second operation mode, the control means to transition the signal processing means to the first operation mode in response to the event signal being generated by the imaging means;
a step of controlling, when the signal processing means is in the first operation mode, the control means to transition the signal processing means to the second operation mode when the event signal is not input from the imaging means within a threshold time;
a step of controlling the control means to transition the signal processing means from the second operation mode to the first operation mode when the event signal is input from the imaging means, and causing the information holding means to hold the event signal input from the imaging means until the signal processing means transitions from the second operation mode to the first operation mode;
13. A method for controlling an information processing apparatus comprising the steps of: A program for causing a computer to function as the control means of the information processing device according to any one of claims 1 to 11 .

Images