JP7632029B2 - Timing adjustment method - Google Patents

Description

The present invention relates to a timing adjustment method.

Image data may be transmitted from an image transmitting device located at at least one base to an image receiving device via a network, and the image transmitted from the image transmitting device may be displayed on a display device provided in the image receiving device. In this case, with the practical application of the fifth generation mobile communication system (5G), image data can be transmitted with low latency, so the delay due to the transmission of image data is negligible.

Japanese Patent Application Publication No. 11-88305

However, delays due to internal processing in the image transmitting device and image receiving device cannot be ignored. In addition, when image transmitting devices are installed at multiple locations and image data is transmitted from the multiple image transmitting devices to the image receiving device, there is variation in the delay time due to internal processing in the multiple image transmitting devices, which can cause the timing of the image displayed on the display device to vary, creating an awkward feeling. There is a need to understand the delay time, including the delay due to internal processing in the image transmitting device and image receiving device, when image data is transmitted from the image transmitting device to the image receiving device via a network.

The present invention aims to provide a timing adjustment method that can adjust the timing at which image data captured of a subject performing a specific action performed separately at a sending location and a reference location is displayed at the reference location to match.

The present invention provides a timing adjustment method in which a camera provided in an image transmitting device located at a transmitting base captures an image of a subject performing the action of playing a musical instrument , generates image data, and transmits the image data via a network to an image receiving device located at a reference base, measures the delay time when the image receiving device displays the received image data on a display device, sets a delay time correction value based on the delay time in a timing indication unit located at the transmitting base, and signals the subject, via the timing indication unit, to inform the subject of the timing to start the action of playing the musical instrument at a time that is the time indicated by the delay time correction value before the time at which the subject has been instructed to start the action of playing the musical instrument .

The timing adjustment method of the present invention can adjust the timing to match when image data captured of a subject performing a specific action performed separately at the sending location and the reference location is displayed at the reference location.

FIG. 1 is a block diagram conceptually showing an image transmission system in which image data is transmitted from a plurality of image transmitting devices to an image receiving device via a network. FIG. 2 is a block diagram showing an example configuration of an image transmitting device and an image receiving device configured to measure the delay time from the time when a camera in the image transmitting device starts capturing a moving image to the time when a display device in the image receiving device starts displaying the moving image. FIG. 3 is a diagram showing the 1 PPS signal and the 10 MHz clock generated by the GNSS satellite wave clock 13 in FIG. 2, and the 148.5 MHz clock generated by the clock generating unit 14. FIG. 4 is a diagram showing the 1 PPS signal and the 10 MHz clock generated by the GNSS satellite wave clock 23 in FIG. 2, and the 148.5 MHz clock generated by the clock generating unit 24. FIG. 5 is a flowchart showing the process executed by the delay time measurement unit 28 in FIG. FIG. 6 is a flowchart showing the detailed process of step S1 in FIG. FIG. 7 is a flowchart showing the detailed process of step S2 in FIG. FIG. 8 is a block diagram showing a delay time management server that manages delay times when image data is transmitted and received bidirectionally between image transmitting/receiving devices disposed at three locations. FIG. 9 is a diagram showing an example of the delay time stored in the delay time management server shown in FIG. FIG. 10 is a block diagram showing an image transmitting/receiving device that adjusts the timing of giving a signal to start a predetermined operation based on the delay time stored in a delay time management server. FIG. 11 is a block diagram showing a specific example of the configuration of the timing instruction unit 29 in FIG. FIG. 12 is a flowchart showing a process executed by the image transmitting/receiving devices arranged at the three locations shown in FIG. 10 to synchronize the reception timing of image data. FIG. 13 is a block diagram showing an image transmitting device configured to correct the timing shift of image data frames, and a delay time management server suitable for correcting the timing shift of image data frames. FIG. 14 is a diagram showing an example of the delay time stored in the delay time management server shown in FIG.

The following describes one or more embodiments of an image transmitting device, an image receiving device, a delay time measuring device, a delay time management server, and a timing adjustment method with reference to the attached drawings.

First, using FIG. 1, a conceptual explanation will be given of an image transmission system that transmits image data from multiple image transmitting devices to an image receiving device via a network. The image data is moving image data. Note that the image transmission system may transmit audio data in addition to image data. The process of transmitting and receiving audio data is not illustrated.

In FIG. 1, image transmission devices 10A to 10C are located at three different bases. An image receiving device 20 is located at a base different from the bases where the image transmission devices 10A to 10C are located. The image transmission devices 10A to 10C each include a camera 11 that captures an image of a subject, and a transmission unit 12 that transmits image data generated by the camera 11 capturing an image of the subject.

Image data transmitted from the image transmitting devices 10A-10C is transmitted to the image receiving device 20 via a network 30. The network 30 is typically the Internet. The image receiving device 20 includes a receiving unit 21 that receives image data, and a display device 22 that displays moving images based on the image data. The display device 22 may simultaneously display the moving images from the image transmitting devices 10A-10C, or may switch between the moving images from the image transmitting devices 10A-10C. An image transmitting device that does not specify which of the image transmitting devices 10A-10C is referred to as an image transmitting device 10.

<Specific configuration examples of the image transmitting device, the image receiving device, and the delay time measuring device>
Fig. 2 shows an example of the configuration of the image transmitting device 10 and the image receiving device 20 configured to measure the delay time from the time when the camera 11 in the image transmitting device 10 starts capturing a moving image to the time when the display device 22 in the image receiving device 20 starts displaying the moving image. The image transmitting device 10 and the image receiving device 20 shown in Fig. 2 have a suitable configuration for grasping the delay time including the delay due to their internal processing.

The delay time measurement unit 28 provided in the image receiving device 20 constitutes a delay time measurement device of one or more embodiments, and measures a more accurate delay time, including delays due to internal processing in the image transmitting device 10 and the image receiving device 20, as described below. The image transmitting device 10 is an image transmitting device of one or more embodiments. The image receiving device 20 is an image receiving device of one or more embodiments.

As shown in FIG. 2, the image transmission device 10 includes a GNSS satellite radio clock 13, a clock generation unit 14, an image memory 15, and a communication unit 16 in addition to the camera 11. The camera 11 has a control unit 111, an image sensor 112, and an image processing circuit 113. The image memory 15 includes a write/read control unit 150. The communication unit 16 functions as the transmission unit 12 in FIG. 1.

In addition to the display device 22, the image receiving device 20 includes a GNSS satellite radio clock 23, a clock generating unit 24, a communication unit 25, an image processing circuit 26, an image memory 27, and a delay time measuring unit 28. The display device 22 has a control unit 221, a drive circuit 222, and a liquid crystal panel 223. The image memory 27 includes a write/read control unit 270. The communication unit 25 functions as the receiving unit 21 in FIG. 1.

The GNSS satellite radio clock 13 receives radio waves from a satellite 40 for the Global Navigation Satellite System (GNSS), and outputs a 1 PPS signal, which is a pulse with a 1 second interval as shown in FIG. 3(a), and a 10 MHz clock (hereinafter, 10 MHz clock) as shown in FIG. 3(b). One example of the GNSS is the Global Positioning System (GPS).

The clock generating unit 14 generates a 148.5 MHz clock (hereinafter, 148.5 MHz clock) shown in FIG. 3(c) based on the input 1 PPS signal and 10 MHz clock, and supplies it to the camera 11, image memory 15, and communication unit 16. For convenience of illustration, the 148.5 MHz clock shown in FIG. 3(c) is shown with a significantly longer cycle of one clock. Note that the clock generating unit 14 may generate clocks of other video synchronization frequencies, such as 297 MHz, 74.25 MHz, 27 MHz, etc., in addition to 148.5 MHz.

The image transmission device 10 may include a clock generation unit that generates a clock including time information based on radio waves received from a GNSS satellite 40. A 10 MHz clock may be supplied to the camera 11, image memory 15, and communication unit 16 to operate the camera 11, image memory 15, and communication unit 16 on the 10 MHz clock. In this case, the GNSS satellite radio clock 13 serves as the clock generation unit that generates a clock including time information. There are no limitations on the frequency of the clock supplied to the camera 11, image memory 15, and communication unit 16.

3A at time 0:00:00, and the next pulse at time 0:00:01. The clock generating unit 14 outputs a 148.5 MHz clock including time information, such as clocks with clock numbers 0 to 148.5×10 ⁻⁶ −1 based on time 0:00:00, and clocks with clock numbers 0 to 148.5×10 ⁻⁶ −1 based on time 0:00:01.

The control unit 111 of the camera 11 controls the electronic shutter in the image sensor 112, and the image sensor 112 captures an image of a subject. The control unit 111 may be a central processing unit provided in the camera 11. Image data generated by the image sensor 112 capturing an image of a subject is subjected to various types of image processing by the image processing circuit 113 and is supplied to the image memory 15.

The image processing in the image processing circuit 113 is one or more of defective pixel interpolation processing, black level processing, white balance processing, demosaic processing, shading correction processing, and compression encoding processing. The image processing in the image processing circuit 113 includes at least compression encoding processing.

The control unit 111 controls the clock number of the 148.5 MHz clock, which indicates the time t0 when the image sensor 112 released the electronic shutter for a specified reference pixel to generate pixel data, to be superimposed as metadata on the blanking period of the image data. The reference pixel is preferably the first pixel in the first row in the frame. In other words, the clock number included in the image data output by the camera 11 indicates the time when the camera 11 started generating image data for each frame. As shown in FIG. 3(c), the clock number indicating the time t0 included in the image data output by the camera 11 is, for example, Cn0, which is clock number 0.

The write/read control unit 150 writes image data to the image memory 15. At this time, the write/read control unit 150 writes image data to the image memory 15 in a state in which the clock number of the 148.5 MHz clock indicating the time t1 at which the pixel data of the reference pixel of each frame was written to the image memory 15 is superimposed as metadata onto the blanking period of the image data. As shown in FIG. 3(c), the clock number indicating the time t1 contained in the image data written to the image memory 15 is assumed to be Cn1. The image data written to the image memory 15 has the clock numbers Cn0 and Cn1 superimposed thereon.

The write/read control unit 150 reads out image data stored in the image memory 15 and supplies it to the communication unit 16. The communication unit 16 packetizes the image data read out from the image memory 15 and transmits it to the image receiving device 20 via the network 30. At this time, the communication unit 16 packetizes the image data with the clock number of the 148.5 MHz clock indicating the time t2 at which pixel data of the reference pixel of each frame is transmitted superimposed as metadata on the blanking period of the image data. As shown in FIG. 3(c), the clock number indicating the time t2 at which pixel data of the reference pixel is transmitted is assumed to be Cn2. As a result, the clock number Cn2 indicates the time t2 at which the pixel data of the reference pixel is transmitted.

The communication unit 16 also adds clock frequency information to the image data and packetizes it to inform the image receiving device 20 that the camera 11, image memory 15, and communication unit 16 are operating at a 148.5 MHz clock. The packet data transmitted from the image transmitting device 10 to the image receiving device 20 via the network 30 includes the image data and clock frequency information, and the image data has clock numbers Cn0, Cn1, and Cn2 superimposed on it.

The GNSS satellite radio clock 23 in the image receiving device 20, like the GNSS satellite radio clock 13, receives radio waves from the satellite 40 and outputs a 1 PPS signal, which is a pulse with a 1 second interval as shown in FIG. 4(a), and a 10 MHz clock as shown in FIG. 4(b).

In FIG. 2, the satellite 40 that transmits radio waves received by the GNSS satellite radio clock 13 and the GNSS satellite radio clock 23 is a single common satellite 40. The satellite 40 that transmits radio waves received by the GNSS satellite radio clock 13 may be different from the satellite 40 that transmits radio waves received by the GNSS satellite radio clock 23. The GNSS satellite radio clocks 13 and 23 may receive radio waves from any of the multiple satellites 40 operated by a single global navigation satellite system.

The communication unit 25 receives packet data including image data and clock frequency information, supplies the clock frequency information to the clock generation unit 24, and supplies the image data to the image processing circuit 26. Since the clock frequency information indicating 148.5 MHz is input to the clock generation unit 24, the clock generation unit 24 generates the 148.5 MHz clock shown in FIG. 4(c) based on the input 1 PPS signal and 10 MHz clock. The 148.5 MHz clock is supplied to the display device 22, the communication unit 25, the image memory 27, and the delay time measurement unit 28.

The 148.5 MHz clock shown in FIG. 4(c) is the same clock (same phase and same frequency) as the 148.5 MHz clock shown in FIG. 3(c).

The clock that the clock generation unit 24 supplies to the display device 22, the communication unit 25, the image memory 27, and the delay time measurement unit 28 is a 148.5 MHz clock that includes time information, as described above.

As shown in FIG. 4(c), the communication unit 25 superimposes the clock number of the 148.5 MHz clock indicating the time t3 when the pixel data of the reference pixel of each frame was received from the network 30 onto the blanking period of the image data as metadata. The clock number indicating the time t3 when the pixel data of the reference pixel was received is assumed to be Cn3. Therefore, the image data output from the communication unit 25 has the clock numbers Cn0, Cn1, Cn2, and Cn3 superimposed on it.

The image data output from the communication unit 25 is subjected to various image processing by the image processing circuit 26 and supplied to the image memory 27. The image processing in the image memory 27 is at least a compression decoding process. The image processing circuit 26 supplies the clock numbers Cn0, Cn1, Cn2, and Cn3 contained in the image data to the delay time measurement unit 28.

The write/read control unit 270 writes the image data supplied to the image memory 27 into the image memory 27, and reads out the image data stored in the image memory 27 and supplies it to the drive circuit 222 of the display device 22. At this time, the write/read control unit 270 supplies the clock number of the 148.5 MHz clock indicating the time t4 at which the pixel data of the reference pixel of each frame was read out to the delay time measurement unit 28. The time t4 is also the time at which the pixel data of the reference pixel was supplied to the drive circuit 222. As shown in FIG. 4(c), the clock number indicating the time t4 at which the image data was read out from the image memory 27 is Cn4.

The control unit 221 of the display device 22 controls the drive circuit 222 so as to display the image data supplied to the drive circuit 222 on the liquid crystal panel 223. The control unit 221 may be a central processing unit provided in the display device 22. The control unit 221 supplies the clock number of the 148.5 MHz clock indicating the time t5 at which the pixel data of the reference pixel of each frame was displayed on the liquid crystal panel 223 to the delay time measurement unit 28. As shown in (c) of FIG. 4, the clock number indicating the time t5 at which the pixel data of the reference pixel was displayed on the liquid crystal panel 223 is assumed to be Cn5.

Note that the display device 22 does not need to operate on a 148.5 MHz clock. The 148.5 MHz clock is supplied to the display device 22 in order to supply the delay time measurement unit 28 with clock number Cn5, which indicates time t5 in the 148.5 MHz clock.

When the display device 22 is a projection type display device rather than a direct-view type display device, the liquid crystal panel 223 is a liquid crystal display element for a projection type display device. The display device 22 may also include a display panel (display element) other than a liquid crystal panel.

The delay time measurement unit 28 calculates the delay time between the image transmitting device 10 and the image receiving device 20 according to the process shown in the flowcharts of Figures 5 to 7, and determines a delay time correction value for correcting the delay due to the delay time.

In FIG. 5, the delay time measurement unit 28 sets the shooting time in the image transmission device 10 in step S1, and sets the display time in the image reception device 20 in step S2. FIG. 6 shows the detailed processing of step S1. In FIG. 6, the delay time measurement unit 28 determines whether or not clock number Cn0 (i.e., time t0) exists in step S11. If clock number Cn0 exists (YES), the delay time measurement unit 28 sets the time t0 indicated by clock number Cn0 as the shooting time in step S13.

If the clock number does not exist in step S11 (NO), the delay time measurement unit 28 determines whether or not the clock number Cn1 (i.e., time t1) exists in step S12. If the clock number Cn1 exists (YES), the delay time measurement unit 28 sets the time t1 indicated by the clock number Cn1 as the shooting time in step S14. If the clock number Cn1 does not exist (NO), the delay time measurement unit 28 sets the time t2 indicated by the clock number Cn2 as the shooting time in step S15.

Figure 7 shows the detailed processing of step S2. In Figure 7, the delay time measurement unit 28 determines whether or not clock number Cn5 (i.e., time t5) exists in step S21. If clock number Cn5 exists (YES), the delay time measurement unit 28 sets time t5 as the display time in step S23.

If clock number Cn5 does not exist in step S21 (NO), the delay time measurement unit 28 determines whether clock number Cn4 (i.e., time t4) exists in step S22. If clock number Cn4 exists (YES), the delay time measurement unit 28 sets time t4 as the display time in step S24. If clock number Cn4 does not exist (NO), the delay time measurement unit 28 sets time t3 indicated by clock number Cn3 as the display time in step S25.

In FIG. 6, the reason for determining whether clock number Cn0 exists is that there may be cases where the camera 11 is not configured to superimpose clock number Cn0, which indicates time t0 when the camera 11 starts generating image data for each frame, onto the image data. Also, the reason for determining whether clock number Cn1 exists is that there may be cases where the camera 11 is not configured to superimpose clock number Cn1, which indicates time t1 when the image memory 15 writes the image data, onto the image data.

In FIG. 7, the presence or absence of clock number Cn5 is determined because the display device 22 may not be configured to supply the delay time measurement unit 28 with clock number Cn5 indicating time t5 at which the image data was displayed on the liquid crystal panel 223. The presence or absence of clock number Cn4 is determined because the display device 22 may not be configured to supply the delay time measurement unit 28 with clock number Cn4 indicating time t4 at which the image memory 27 read out the image data.

Returning to FIG. 5, in step S3, the delay time measurement unit 28 calculates the delay time between the image transmitting device 10 and the image receiving device 20, which is the difference between the shooting time and the display time. In step S4, the delay time measurement unit 28 determines whether the shooting time is time t2 and the display time is time t3. If the shooting time is time t2 and the display time is time t3, the time from time 2 to time t3 is the actual transmission time of the image data over the network 30. Therefore, it does not represent the delay time from when the image transmitting device 10 starts generating the image data to when the image receiving device 20 starts displaying the image data.

If the shooting time is time t2 and the display time is time t3 in step S4 (YES), the delay time measurement unit 28 determines in step S5 whether a pre-measured fixed correction value is set in the delay time measurement unit 28. The fixed correction value is the sum of either the average time from time t0 to time t2 or the average time from time t3 to time t5, or both.

If a fixed correction value measured in advance is set in step S5 (YES), the delay time correction value is determined by adding the fixed correction value to the delay time obtained in step S3, and the process ends.

If in step S4 the shooting time is not time t2 and the display time is not time t3 (NO), or if in step S5 the previously measured fixed correction value is not set (NO), in step S7 the delay time measurement unit 28 determines the delay time obtained in step S3 as the delay time correction value as is, and ends the process.

As described above, the image transmission device 10 includes the camera 11, the clock generation unit 14, the image memory 15, and the communication unit 16. The clock generation unit 14 generates a first clock (a 148.5 MHz clock, for example) that includes time information based on radio waves received from a GNSS satellite 40. The camera 11 operates according to the first clock, and generates image data by the image sensor 112 capturing an image of a subject. The image memory 15 operates according to the first clock, and writes and reads image data under the control of the write/read control unit 150.

The communication unit 16 transmits image data that is read from the image memory 15 and has at least one of the first to third clock numbers superimposed thereon. The first to third clock numbers are clock numbers Cn0 to Cn2.

The first clock number indicates the time at which the image sensor 112 of the camera 11 generated pixel data for a reference pixel in a frame. If the camera 11 is configured to execute a process of superimposing the first clock number on the image data, image data including the first clock number is transmitted.

The second clock number indicates the time when the write/read control unit 150 wrote the pixel data of the reference pixel to the image memory 15. If the write/read control unit 150 is configured to execute a process of superimposing the second clock number on the image data, image data including the second clock number is transmitted. The third clock number indicates the time when the communication unit 16 transmitted the pixel data of the reference pixel. If the communication unit 16 is configured to execute a process of superimposing the third clock number on the image data, image data including the third clock number is transmitted.

The image receiving device 20 includes a communication unit 25 that receives image data on which at least one of the first clock number, the second clock number, and the third clock number is superimposed. If the communication unit 25 is considered to be the first communication unit, the communication unit 16 is the second communication unit. If the communication unit 16 is considered to be the first communication unit, the communication unit 25 is the second communication unit.

The first clock number is a clock number in the first clock generated by the clock generation unit 14 (first clock generation unit) and indicates the time when the image sensor 112 of the camera 11 generated pixel data for a reference pixel in a frame. The second clock number is a clock number in the first clock and indicates the time when the write/read control unit 150 (first write/read control unit) wrote the pixel data for the reference pixel to the image memory 15 (first image memory). The third clock number is a clock number in the first clock and indicates the time when the communication unit 16 (second communication unit) transmitted the pixel data for the reference pixel.

The image receiving device 20 further comprises a clock generating unit 24 (second clock generating unit), an image memory 27 (second image memory), a display device 22, and a delay time measuring unit 28. The clock generating unit 24 generates a second clock that is the same as the first clock. The image memory 27 operates according to the second clock, and writes and reads image data based on the control of a write/read control unit 270 (second write/read control unit). The display device 22 displays the image data read from the image memory 27.

The delay time measurement unit 28 measures the delay time indicating the difference between any one of the first to third clock numbers and any one of the fourth to sixth clock numbers. The fourth clock number is a clock number in the second clock and indicates the time when the communication unit 25 received the pixel data of the reference pixel. The fifth clock number is a clock number in the second clock and indicates the time when the write/read control unit 270 reads the pixel data of the reference pixel from the image memory 27. The sixth clock number is a clock number in the second clock and indicates the time when the display device 22 displays the pixel data of the reference pixel.

When the delay time measurement unit 28 acquires the first clock number (clock number Cn0), it is preferable to measure the delay time using the first clock number. When the delay time measurement unit 28 is unable to acquire the first clock number and acquires the second clock number (clock number Cn1), it is preferable to measure the delay time using the second clock number. When the delay time measurement unit 28 acquires the sixth clock number (clock number Cn5), it is preferable to measure the delay time using the sixth clock number. When the delay time measurement unit 28 is unable to acquire the sixth clock number and acquires the fifth clock number (clock number Cn4), it is preferable to measure the delay time using the fifth clock number.

The delay time measurement unit 28 is a delay time measurement device of one or more embodiments. As described above, the delay time measurement unit 28 determines the difference between any one of the first to third clock numbers and any one of the fourth to sixth clock numbers. In this way, the delay time measurement unit 28 measures the delay time when the image receiving device 20 receives and displays image data generated by the image transmitting device 10 and transmitted to the image receiving device 20.

According to one or more embodiments of the image transmitting device, image receiving device, and delay time measuring device, it is possible to grasp a more accurate delay time when transmitting image data from the image transmitting device 10 to the image receiving device 20 via the network 30.

<Delay time management server>
8, image transmitting/receiving devices 120A to 120C are disposed at bases B1 to B3, respectively. The image transmitting/receiving devices 120A to 120C communicate with each other bidirectionally via a network 30. An image transmitting/receiving device that does not specify which of the image transmitting/receiving devices 120A to 120C is referred to as an image transmitting/receiving device 120. The image transmitting/receiving device 120 has the configurations of both the image transmitting device 10 and the image receiving device 20 shown in FIG.

As shown in FIG. 8, the image transmission/reception device 120 generally comprises a camera 11, a display device 22, a delay time measurement unit 28, a control unit 122, and a transmission/reception unit 121. Although not shown in FIG. 8, the image transmission/reception device 120 also comprises a GNSS satellite radio clock 13 (or 23) and a clock generation unit 14 (or 24), and a 148.5 MHz clock is supplied to the camera 11 and the display device 22. The transmission/reception unit 121 corresponds to the communication units 16 and 25. The control unit 122 corresponds to the control units 111 and 221.

The image transmitting device 10 shown in FIG. 2 is a detailed description of the configuration when the image transmitting/receiving device 120 shown in FIG. 8 is operating as an image transmitting device. The image receiving device 20 shown in FIG. 2 is a detailed description of the configuration when the image transmitting/receiving device 120 shown in FIG. 8 is operating as an image receiving device.

A delay time management server 50 is connected to the network 30. The delay time management server 50 communicates bidirectionally with the image transmitting/receiving devices 120A to 120C. As shown in FIG. 9, the delay time management server 50 stores the delay time when the image transmitting/receiving devices 120A to 120C located at bases B1 to B3 are the transmitting side and the other image transmitting/receiving devices 120 are the receiving side. In FIG. 9, the delay time is listed together with information indicating from which time among times t0 to t2 the delay time extends to which time among times t3 to t5.

The delay time management server 50 may update the delay time shown in FIG. 9 each time the delay time measurement unit 28 of the image transmission/reception device 120A to 120C measures the delay time. The delay time management server 50 may update the delay time shown in FIG. 9 to the latest delay time measured by the delay time measurement unit 28 each time a predetermined time has elapsed. The image transmission/reception device 120 can read the delay time stored in the delay time management server 50.

As described above, one or more delay time management servers, ie, delay time management servers 50, are connected to the network 30. At least two image transmitting/receiving devices 120, ie, a first and a second image transmitting/receiving device, are connected to the network 30 so as to communicate with each other bidirectionally. Any two image transmitting/receiving devices 120 among the image transmitting/receiving devices 120A to 120C are the first and second image transmitting/receiving devices.

The first image transmission/reception device includes a first delay time measurement unit (delay time measurement unit 28) that measures a first delay time when the first image transmission/reception device receives and displays image data generated by the second image transmission/reception device and transmitted to the first image transmission/reception device. The second image transmission/reception device includes a second delay time measurement unit (delay time measurement unit 28) that measures a second delay time when the second image transmission/reception device receives and displays image data generated by the first image transmission/reception device and transmitted to the second image transmission/reception device.

The delay time management server 50 stores a first delay time transmitted from a first image transmission/reception device via the network 30 and a second delay time transmitted from a second image transmission/reception device via the network 30.

According to one or more embodiments of the delay time management server 50, it is possible to more accurately manage the delay time when transmitting image data from an image transmitting/receiving device 120 operating as an image transmitting device to an image transmitting/receiving device 120 operating as an image receiving device via the network 30.

<Method of adjusting timing when receiving image data>
10, a musical instrument is played at each of the locations B1 to B3, and the image transmitting/receiving device 120B at the location B2, where the performance supervisor is located, receives image data transmitted from the image transmitting/receiving devices 120A and 120C. Playing a musical instrument is an example of a predetermined action performed by a person who is a subject. The subject who plays a musical instrument is a performer.

The camera 11 of the image transmitting/receiving device 120A photographs the performers at location B1, and the transmitting/receiving unit 121 transmits the image data to the image transmitting/receiving device 120B. The camera 11 of the image transmitting/receiving device 120C photographs the performers at location B3, and the transmitting/receiving unit 121 transmits the image data to the image transmitting/receiving device 120B. The camera 11 of the image transmitting/receiving device 120B photographs the performers at location B2, and the display device 22 displays a video. The display device 22 of the image transmitting/receiving device 120B may simultaneously display the video generated by the image transmitting/receiving devices 120A to 120C, or may switch between the video generated by the image transmitting/receiving devices 120A to 120C.

The image transmission/reception devices 120A to 120C are equipped with a timing instruction unit 29 for signaling the start of performance to the performers located at the respective bases B1 to B3. As shown in FIG. 11, the timing instruction unit 29 has a control unit 291, a clock 292, an operation unit 293, a delay time correction value storage unit 294, and a display unit 295. The clock 292 may be the GNSS satellite radio clock 13 (or 23). The display unit 295 may be a liquid crystal panel or an LED.

The timing instruction unit 29 is configured so that when the time to start playing is set in the control unit 291 by the operation unit 293, the control unit 291 monitors the time measured by the clock 292 and displays a signal indicating the timing to start playing on the display unit 295. For example, if playing starts at each of the locations B1 to B3 at 13:00:00, there is the above-mentioned delay time until the image data is transmitted from the locations B1 and B3 to the location B2 and displayed on the display device 22, so the timing of the playing at all the locations B1 to B3 displayed at the location B2 will not be the same.

Then, the transmission/reception unit 121, under the control of the control unit 122, reads the delay time shown in FIG. 9 from the delay time management server 50 and supplies it to the control unit 291 of the timing instruction unit 29. The control unit 291 causes the delay time correction value storage unit 294 to store a delay time correction value based on the delay time. Here, an example is shown in which the delay time is used as the delay time correction value as is.

In the example shown in FIG. 10, the delay time correction value storage unit 294 of the timing instruction unit 29 provided in the image transmission/reception device 120A stores a delay time of 10.3 ms with location B1 as the sending side and location B2 as the receiving side. The delay time correction value storage unit 294 of the timing instruction unit 29 provided in the image transmission/reception device 120C stores a delay time of 10.2 ms with location B3 as the sending side and location B2 as the receiving side.

The delay time correction value storage unit 294 of the timing instruction unit 29 of the image transmission/reception device 120B does not need to store a delay time correction value. In FIG. 10, since no delay time is supplied from the transmission/reception unit 121 to the timing instruction unit 29, the connection between the transmission/reception unit 121 and the timing instruction unit 29 is indicated by a dashed arrow line.

The control unit 291 of the timing indication unit 29 provided in the image transmission/reception device 120A displays on the display unit 295 a cue indicating the timing to start performance at a time 10.3 ms before the delay time of 13:00:00, which is the time to start performance. The control unit 291 of the timing indication unit 29 provided in the image transmission/reception device 120C displays on the display unit 295 a cue indicating the timing to start performance at a time 10.2 ms before the delay time of 13:00:00, which is the time to start performance.

The control unit 291 of the timing indicator 29 of the image transmission/reception device 120B displays a signal on the display unit 295 indicating the timing to start playing at 13:00:00, which is the time to start playing.

The performers at each of the locations B1 to B3 start playing in sync with the signal displayed on the display unit 295 of the timing indicator 29 at each of the locations B1 to B3. As a result, the image data displayed on the display device 22 at the location B2 is displayed in a state where the timing of the performances at all of the locations B1 to B3 is synchronized.

The process executed by the image transmitting/receiving devices 120A to 120C to synchronize the reception timing of image data will be described using the flowchart shown in FIG. 12. In FIG. 12, the control unit 122 of the image transmitting/receiving device 120B sets the image transmitting/receiving device 120B as the reference location in step S31. The control units 122 of the image transmitting/receiving devices 120A and 120C, which are the transmitting side locations, obtain the corresponding delay time from the delay time management server 50 in step S32.

In step S33, the control unit 291 of the image transmission/reception devices 120A and 120C sets the delay time as the delay time correction value in the delay time correction value storage unit 294. Here, the delay time is treated as the delay time correction value as is. In step S34, the control unit 291 of the image transmission/reception devices 120A to 120C sets the performance start time based on the operation input from the operation unit 293.

In step S351, the control unit 122 of the image transmitting/receiving devices 120A and 120C, which are the transmitting bases, sets the performance start timing to a time that is earlier than the performance start time by the delay time correction value. In step S352, the control unit 122 of the image transmitting/receiving device 120B, which is the reference base, sets the performance start timing to the performance start time.

In step S361, the control unit 122 of the image transmission/reception devices 120A and 120C determines whether the current time has reached the performance start timing. If the current time has not reached the performance start timing (NO), the control unit 122 repeats the process of step S361. If the current time has reached the performance start timing (YES), the control unit 122 transitions the process to step S371.

In step S362, the control unit 122 of the image transmission/reception device 120B determines whether the current time has reached the performance start timing. If the current time has not reached the performance start timing (NO), the control unit 122 repeats the process of step S362. If the current time has reached the performance start timing (YES), the control unit 122 transitions the process to step S372.

In step S371, the control unit 122 of the image transmission/reception device 120A and 120C controls the display unit 295 to display a cue notifying the start of performance at the performance start timing, and ends the process. In step S372, the control unit 122 of the image transmission/reception device 120B controls the display unit 295 to display a cue notifying the start of performance at the performance start timing, and ends the process.

As described above, in the image transmission system shown in FIG. 10, the image transmitting device (image transmitting/receiving device 120A or 120C) located at the transmitting side base includes a camera 11. The image transmitting device transmits image data generated by the camera 11 by capturing an image of a subject performing a predetermined action to the image receiving device (image transmitting/receiving device 120B) located at the reference base via the network 30.

The timing adjustment method executed in the image transmission system measures the delay time when the image receiving device displays the received image data on the display device 22. The timing adjustment method sets a delay time correction value based on the delay time in a timing instruction unit 29 arranged at the transmitting side base. The timing adjustment method causes the timing instruction unit 29 to signal the subject to inform the subject of the timing to start the specified action at a time indicated by the delay time correction value before the time the subject is instructed to start the specified action.

According to one or more embodiments of the timing adjustment method, it is possible to adjust the timing at which image data captured of a subject performing a specific action performed separately at the transmitting location and the reference location is displayed at the reference location to match.

<Method of adjusting frame timing when receiving image data>
Even if the timing of the start of performance is adjusted using the configuration shown in FIG. 10, the timing of the frames of image data generated by the image transmitting/receiving devices 120A to 120C is shifted, and image noise may occur when switching between the moving images generated by the image transmitting/receiving devices 120A to 120C for display.

Figure 13 shows an image transmitting/receiving device 120 configured to correct the timing shift of image data frames, and a delay time management server 50 suitable for correcting the timing shift of image data frames. In Figure 13, similar to Figure 10, image transmitting/receiving device 120B receives image data transmitted from image transmitting/receiving devices 120A and 120C.

Image transmission/reception device 120B has a vertical synchronization delay circuit 123. Image data transmitted from image transmission/reception devices 120A and 120C and received by transmission/reception unit 121, and image data output from camera 11 are input to vertical synchronization delay circuit 123. It is assumed that the vertical synchronization signals of the image data transmitted from image transmission/reception devices 120A and 120C are delayed with respect to the vertical synchronization signal of the image data output from camera 11.

The vertical synchronization delay circuit 123 delays the vertical synchronization of the image data output from the camera 11 to reduce the deviation of the vertical synchronization signal of the image data transmitted from the image transmission/reception devices 120A and 120C relative to the vertical synchronization signal of the image data output from the camera 11. If the deviation amounts of the vertical synchronization signals of the image data transmitted from the image transmission/reception devices 120A and 120C are different, the vertical synchronization delay circuit 123 delays the vertical synchronization of the image data with the smaller deviation amount to match the image data with the larger deviation amount. The vertical synchronization delay circuit 123 delays the image data from the camera 11 to match the deviation amount, thereby matching the vertical synchronization of the three image data.

The vertical synchronization delay circuit 123 supplies the transmission/reception unit 121 with a first delay time of the vertical synchronization signal of the image data transmitted from the image transmission/reception device 120A and a second delay time of the vertical synchronization signal of the image data transmitted from the image transmission/reception device 120C, relative to the vertical synchronization signal of the image data output from the camera 11. The vertical synchronization delay circuit 123 functions as a vertical synchronization deviation measurement unit that measures the first and second delay times.

The transmitter/receiver 121 transmits the first and second delay times to the delay time management server 50 via the network 30 under the control of the controller 122. Assume that the first delay time is 3.2 ms and the second delay time is 0.3 ms.

As shown in FIG. 14, the delay time management server 50 stores a delay time obtained by adding a first delay time of 3.2 ms to a delay time of 10.3 ms when base B1 is the sending side and base B2 is the receiving side. The delay time management server 50 also stores a delay time obtained by adding a second delay time of 0.3 ms to a delay time of 10.2 ms when base B3 is the sending side and base B2 is the receiving side.

The delay time correction value holding unit 294 of the timing instruction unit 29 in the image transmission/reception devices 120A and 120C holds the delay times (10.3 + 3.2) ms and (10.2 + 0.3) ms as delay time correction values, respectively.

The control unit 291 of the timing instruction unit 29 in the image transmission/reception device 120A displays on the display unit 295 a cue informing the user that it is time to start playing at a time before the performance start time, which is the delay time correction value (10.3 + 3.2) ms. The control unit 291 of the timing instruction unit 29 in the image transmission/reception device 120C displays on the display unit 295 a cue informing the user that it is time to start playing at a time before the performance start time, which is the delay time correction value (10.2 + 0.3) ms.

As a result, not only is the image data displayed on the display device 22 at site B2 displayed with the timing of all the performances at sites B1 to B3 synchronized, but the timing discrepancies between the frames of image data generated by the image transmitting/receiving devices 120A to 120C are also corrected.

If the vertical synchronization signal of the image data transmitted from the image transmission/reception device 120A or 120C becomes out of sync with the vertical synchronization signal of the image data output from the camera 11 due to the passage of time, the vertical synchronization delay circuit 123 can delay the vertical synchronization of the image data output from the camera 11 to reduce the deviation of the vertical synchronization signal.

Furthermore, the first delay time of the vertical synchronization signal of the image data transmitted from the image transmitting/receiving device 120A and the second delay time of the vertical synchronization signal of the image data transmitted from the image transmitting/receiving device 120C relative to the vertical synchronization signal of the image data output from the camera 11 are transmitted to the delay time management server 50 and updated to the latest first and second delay times. Therefore, the timing of the frames of image data generated by the image transmitting/receiving devices 120A to 120C is always corrected to an optimal state at predetermined time intervals.

As described above, in the image transmission system shown in FIG. 13, an image transmitting device (image transmitting/receiving device 120A or 120C) equipped with a first camera (camera 11) and an image receiving device equipped with a second camera (camera 11) and a display device 22 are connected via a network 30. The image receiving device (image transmitting/receiving device 120B) is equipped with a delay time measuring unit 28 and a vertical synchronization delay circuit 123 that functions as a vertical synchronization deviation amount measuring unit.

The delay time measurement unit 28 measures the delay time when the first image data generated by the first camera and transmitted by the image transmission device to the image reception device is received by the image reception device and displayed on the display device 22. The vertical synchronization delay circuit 123 measures the vertical synchronization deviation amount indicating the time by which vertical synchronization is deviated between the first image data and the second image data generated by the second camera and displayed on the display device 22. The delay time management server 50 stores the delay time and vertical synchronization deviation amount transmitted from the image reception device via the network 30.

The image transmission system shown in FIG. 13 transmits first image data generated by a first camera provided in an image transmission device located at a transmission side base by capturing an image of a first subject performing a predetermined action to an image reception device located at a reference base via a network 30. The timing adjustment method executed in the image transmission system measures the delay time when the first image data received by the image reception device is displayed on a display device 22.

The timing adjustment method executed in the image transmission system measures a vertical synchronization deviation amount indicating the time by which vertical synchronization is deviated between the first image data and the second image data generated by a second camera provided in the image receiving device by photographing a second subject performing a predetermined operation. The timing adjustment method executed in the image transmission system sets a delay time correction value based on a delay time obtained by adding the delay time and the vertical synchronization deviation amount in a timing instruction unit 29 located at the transmitting side base.

The timing adjustment method executed in the image transmission system involves a timing instruction unit 29 signaling the first subject to inform the first subject of the timing to start a predetermined action at a time indicated by the delay time correction value prior to the time at which the first subject is instructed to start the predetermined action.

According to the delay time management server of one or more embodiments, it is possible to more accurately manage the delay time and the vertical synchronization deviation amount when transmitting image data from the image transmitting/receiving device 120 operating as an image transmitting device to the image transmitting/receiving device 120 operating as an image receiving device via the network 30. According to the timing adjustment method of one or more embodiments, it is possible to adjust the timing, including the timing of the frames of the image data, to match when image data captured of a subject performing a predetermined action performed separately at the transmitting base and the reference base is displayed at the reference base.

The present invention is not limited to one or more of the embodiments described above, and various modifications are possible without departing from the spirit of the present invention.

10, 10A to 10C Image transmitting device 11 Camera 12 Transmitter 13, 23 GNSS satellite radio clock 14, 24 Clock generator 15, 27 Image memory 16, 25 Communication unit 20 Image receiving device 21 Receiving unit 22 Display device 26, 113 Image processing circuit 28 Delay time measuring unit 29 Timing instruction unit 30 Network 40 Satellite 50 Delay time management server 111, 122, 221 Control unit 112 Image sensor 120A to 120C Image transmitting/receiving device 121 Transmitter/receiver unit 123 Vertical synchronization delay circuit (vertical synchronization deviation measuring unit)
150, 270 Write/read control unit 222 Drive circuit 223 Liquid crystal panel

Claims (1)

a camera provided in an image transmitting device arranged at a transmitting side site captures an image of a subject performing an action of playing a musical instrument , the image data being generated by the camera being transmitted to an image receiving device arranged at a reference site via a network, and a delay time when the image receiving device displays the received image data on a display device is measured;
A delay time correction value based on the delay time is set in a timing instruction unit disposed in the transmitting side base,
the timing instruction unit issues a signal to the subject to inform the subject of the timing to start the action of playing the musical instrument at a time that is a time indicated by the delay time correction value before a time at which the subject is instructed to start the action of playing the musical instrument .

Images