WO2016020979A1 - Computer system and intermediary device - Google Patents

Abstract

This computer system comprises an intermediary device which is provided between a storage controller and a server, and which acts as an intermediary for information unit communication (communication according to PCIe). A first device (either the server or the storage controller) has a plurality of first queues, and the intermediary device has a second queue. A first index and a second index are provided for each of the plurality of first queues. Each first index is a value indicating the number of entries stored in the respective first queue, and each second index is a value indicating the number of entries constituting each information unit stored in the respective first queue. When a first index indicates the presence of an entry or entries in the respective first queue, the intermediary device fetches, from the first queue, the number of entries indicated by the second index for the first queue, stores the fetched entries in the second queue, and allocates the entries stored in the second queue to a selected one or ones of a plurality of processors, one or more information units at a time.

Description

Computer system and mediation device

The present invention generally relates to queue control in a computer system.

Generally, in a computer system having a server device (hereinafter referred to as a server) and a storage device (hereinafter referred to as storage), a server device and a storage device manufactured by different vendors are connected to a communication network (for example, an FC (Fibre Channel) network or an IP). (Internet Protocol)).

However, for performance improvement or other purposes, it is conceivable to connect the server and the storage so that they can communicate with each other using the same communication interface as the communication interface used inside. As such a communication interface, for example, PCIe (PCI-Express) is conceivable. For example, Non-Patent Documents 1 and 2 disclose techniques related to PCIe. Specifically, for example, Non-Patent Document 1 discloses a definition of a ring queue for information transfer between a PQI (PCIe architecture Queuing Interface) host and a PQI device. Non-Patent Document 2 discloses the definition of the protocol layer.

PCIe architecture Queuing Interface (PQI) (http://www.t10.org/disk/sop-agnd.htm) SCSI over PCIe architecture (SOP) (http://www.t10.org/disk/sop-agnd.htm)

In the communication between the server and the storage controller, the information unit is usually communicated. The information unit is, for example, a request. For communication between the server and the storage controller, for example, a plurality of queues are provided in at least one of the server and the storage, and information units to be communicated are respectively stored in the plurality of queues. Neither of Non-Patent Documents 1 and 2 discloses a queue control for efficiently handling a plurality of queues for communication between a server and a storage controller.

An intermediary device that mediates communication of information units (communication according to PCIe) is provided between the storage controller and the server. The first device (one of the server and the storage controller) has a plurality of first queues, and the mediation device has a plurality of processors and a second queue. A first and a second index are provided for each of the plurality of first queues. The first index is a value indicating the number of stored entries, and the second index is a value indicating the number of entries constituting the information unit stored in the first queue. If the second index indicates 1 for the first queue corresponding to the first index indicating that the entry exists, the mediation device is the standard length information unit including the entry to be fetched. If the second index indicates any integer greater than or equal to 2, it is determined that the information unit including the fetch target entry is an extended-length information unit, and the number of entries indicated by the second index Is fetched from the first queue, the fetched entry is stored in the second queue, and the entry stored in the second queue is allocated to a processor selected from a plurality of processors in units of information units. The processor performs processing for transferring the information unit constituted by the allocated entry to the second device (the other of the server and the storage controller).

-Multiple queues can be efficiently handled for communication between the server and the storage controller.

1 shows a configuration of a computer system according to an embodiment. Indicates the configuration of ASIC (Application Specific Integrated Circuit). It is a schematic diagram of queue control. The flow of storing the information unit in the first queue is shown. The flow of fetching entries from the first queue to the second queue is shown. Indicates the mode register. A part of the flow of I / O processing is shown. The rest of the flow of I / O processing is shown.

One embodiment will be described with reference to the drawings.

In the following description, the same reference numerals are assigned to a plurality of similar elements, and reference numerals are used when the elements are not distinguished (for example, the queue control unit 230), and these elements are distinguished. In this case, the identification number assigned to the element may be used (for example, queue control unit # 0).

FIG. 1 shows a configuration of a computer system according to the embodiment.

The computer system 101 includes at least one server 123, at least one storage controller (hereinafter referred to as CTL) 181 and at least one ASIC 111 (Application Specific Integrated Circuit). In FIG. 1, a plurality of servers 123, a plurality of ASICs 111, and one CTL 181 are shown. There is one ASIC 111 per server 123. The ASIC 111 is an example of a mediation device. The ASIC 111 is an end point connected to the root complex 125 in the server 123 and the root complex 167 in the CTL 181. As a result, in a system in which the server 123 and the CTL 181 operate as root complexes, the ASIC 111 is viewed as an endpoint from both sides, thereby enabling communication between root complexes.

The server 123 is mounted on the server chassis 121. The computer system 101 has at least one server chassis 121. The server chassis 121 includes two or more servers 123 and a PCIe-MUX (multiplexer) 112. The PCIe-MUX 112 switches the connection between the two or more ASICs 111 and the CTL 181. An ASIC 111 is connected to each server 123, and a PCIe-MUX 112 is connected to the ASIC 111 of each server 123. The connection (communication) between the PCIe-MUX 112 and the CTL 181 is a connection (communication) according to PCIe. Each server 123 and CTL 181 can transmit and receive an I / O (Input / Output) request (an example of an information unit) via the ASIC 111 and the PCIe-MUX 112.

The CTL 181 is mounted on the storage chassis 141. The storage chassis 141 has at least one storage device (for example, a nonvolatile storage device) 123 in addition to the CTL 181. Each storage device 123 is, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive). A storage system including a CTL 181 and one or more storage devices 123 is constructed in the storage chassis 141.

The CTL 181 includes a front end interface (FE-I / F) 151, a back end interface (BE-I / F) 153, a memory 161, and a processor (hereinafter referred to as a storage processor) 163 connected thereto. The FE-I / F 151 is an interface device for communicating with a device on the front end side (the server 123 in this embodiment). The BE-I / F 153 is an interface device for communicating with a device on the back end side (the storage device 123 in this embodiment). The memory 161 has an area for storing control information for controlling the operation of the CTL 181 and a cache area for temporarily storing data input / output to / from the storage device 123. The storage processor 163 controls the operation of the CTL 181 by executing a computer program (for example, an operating system and firmware).

FIG. 2 shows the configuration of the ASIC 111.

The ASIC 111 includes a SW 201, a first I / F 250, a second I / F 240, a mode register 204, a plurality of queue control units 230, a plurality of cores (an example of a processor) 210, and a plurality of DMACs (Dynamic Memory Access Controllers). 220). A first I / F 250, a mode register 204, a second I / F 240, a plurality of queue control units 230, and a plurality of DMACs 220 are connected to the SW 201, and the SW 201 switches connection between these devices. A plurality of cores 210 are interposed between the queue control unit 230 and the DMAC 220. Each queue control unit 230 is connected to any of the plurality of cores 210. Each core 210 is connected to a part of the plurality of DMACs 220 (for example, two DMACs 220). Each core 210 executes firmware, for example.

The first I / F 250 is an interface device (for example, a port) connected to the server 123. The second I / F 240 is an interface device (for example, a port) connected to the CTL 181. In the present embodiment, there is one first I / F 250, and there are two second I / Fs 240. For example, the information unit fetched by the queue control unit 230 from the server 123 via the first I / F 250 is duplexed by the queue control unit 230, and the duplexed information units (two information units) are their information. It may be transferred to the CTL 181 via the two second I / Fs 240 by the core 210 to which the unit is allocated.

The queue control unit 230 performs queue control. The core 210 performs processing for transferring information units. The DMAC 220 performs DMA transfer of information units. The mode register 204 stores a mode value that is a value for a server or a CTL for each of at least one queue control unit 230 of the plurality of queue control units 230. By updating the mode value, the queue control unit 230 corresponding to the mode value can be switched between a server and a CTL. Each of these processes will be described later.

FIG. 3 is a schematic diagram of queue control.

In the following description, the transmission source of the information unit which is one of the server 123 and the CTL 181 is referred to as “first device”, and the transmission destination device of the information unit is referred to as “second device”. Any server 123 and any CTL 181 may be either the first device or the second device. In the example illustrated in FIG. 3, the first device is the CTL 181 and the second device is the server 123. When the first device is the server 123, the second device may be the CTL 181 or another server 123. In an environment having a plurality of CTLs 181, when the first device is the CTL 181, the second device may be another CTL 181.

Each queue control unit 230 in the ASIC 111 corresponds to one first device. For example, the queue control unit 230 # 0 in FIG. 3 corresponds to the CTL 181 and another queue control unit 230 # 3 (see FIG. 2) may correspond to the server 123 # 0 (see FIG. 1). .

The first device has a plurality of first queues 351 and a plurality of notification indexes (denoted as “C index 353” in FIG. 3) corresponding to the plurality of first queues 351, respectively. Each first queue 351 and each C index 353 may be provided on, for example, a storage area (for example, a memory) included in the first device.

The information device to be transmitted is stored in the first queue 351 selected from the plurality of first queues 351 in units of entries by the first device. In the present embodiment, an “entry” is an element obtained by dividing an information unit. The entry length is a predetermined length (for example, 64 bytes). The number of entries constituting the information unit may be 1 or may be 2 or more. For example, the information unit from the server 123 to the CTL 181 has two transfer destination addresses for duplication. Therefore, the length of the information unit is longer (larger in size) than the standard length information unit. It can be an information unit. On the other hand, the information unit from the CTL 181 to the server 123 can be a standard length information unit because it has one transfer destination address. According to this example, the length of the information unit depends on the type of the transmission source device (first device) and the type of the transmission destination device (second device), but instead of or in addition to that viewpoint, The length of the information unit may vary depending on other aspects (for example, the type of information unit). When a plurality of entries are obtained from one information unit, the first device stores the plurality of entries in the selected first queue 351 successively. “Continuously storing a plurality of entries obtained from one information unit” means that the plurality of entries are stored in the selected first queue 351 until the storage is completed. This means that the information unit entry is not stored in the first queue 351. Therefore, in the first queue 351, entries constituting the same information unit are continuous. Note that the length of the standard length information unit is equal to or shorter than the entry length, and may be a length that can accommodate an argument of a frequently used type I / O command. The length of the extended length information unit may be N times the entry length (N is an integer of 2 or more).

The C index 353 indicates a value related to the entry fetched from the first queue 351 corresponding to the C index 353. The C index 353 is used when an entry has been fetched from the first queue 351 corresponding to the C index 353 by the queue control unit 230 or when an entry has been stored in the second queue 371 described later (for example, the entry is information The queue control unit 230 updates the data when it is fetched or stored in units. That is, the update of the C index 353 corresponds to a notification that the entry (information unit) has been fetched or stored from the first queue 351 corresponding to the C index 353. When the first device recognizes the update of the C index 353, the first device may return the C index 353 to a predetermined value. It should be noted that at least one of the plurality of C indexes 353 is replaced with the first device instead of the first device, for example, the ASIC 111 (for example, the first device) interposed between the second device and the first device. May exist in the queue control unit 230) corresponding to.

The queue control unit 230 includes a base address register 372, a storage index (indicated as “P index” in FIG. 3) 361, and a difference index for each first queue 351 included in the first device corresponding to the queue control unit 230. (Denoted as “D index” in FIG. 3) 363. These registers and indexes may be provided in a storage area (for example, a memory) included in the queue control unit 230. The base address register 372 indicates the address of the corresponding first queue 351. The P index 361 is an example of a first index, and is a value indicating a difference from the pointer indicated by the base address register 372 of the corresponding first queue 351, in other words, an entry stored in the corresponding first queue 351. Is a value indicating the number of. The D index 363 is an example of a second index, and is a value indicating the number of entries (the number of entries stored consecutively) constituting the information unit stored in the corresponding first queue 351. At least one of the plurality of P indexes 361 and the plurality of D indexes 363 corresponding to the plurality of first queues 351 is placed in a place other than the queue control unit 230, for example, the first device, instead of the queue control unit 230. May be present. Further, a plurality of D indexes 363 may exist for each first queue 351, and the sum of a plurality of values indicated by the plurality of D indexes 363 is the number of entries in the first queue 351, that is, the P index 361. The value indicated by The D index 363 may be realized as a ring buffer, for example. Further, when the number of entries in the first queue 351 is known from the values of the plurality of D indexes 363, the P index 361 may be omitted.

The queue control unit 230 further includes an entry length register 373 indicating the entry length, and a depth register 375 indicating the depth of the first queue 351 (the maximum number of entries that can be stored in the first queue 351). In the present embodiment, the entry length register 373 and the depth register 375 may be common to the plurality of first queues 351. This is because the length of each entry is the same, and the depth of each first queue 351 is the same. If the entry length and the depth of the first queue 351 are different for each first queue 351, the queue control unit 230 may have an entry length register 373 and a depth register 375 for each first queue 351. The value of the entry length register 373 and the value of the depth register 375 may each be a fixed value or a variable value.

The queue control unit 230 can select the first queue 351 that is the fetch source based on the relationship between the value indicated by the P index 361 and the value indicated by the depth register 375. For example, the queue control unit 230 can preferentially fetch entries from the first queue 351 where the difference between the value indicated by the P index 361 and the value indicated by the depth register 375 is the smallest. Further, the queue control unit 230 can fetch information of a size as indicated by the entry length register 373, that is, an entry from the first queue 351. Further, the queue control unit 230 can specify the location from which the entry is fetched for each first queue 351 from the value indicated by the base address register 372 and the value indicated by the P index 361.

The queue control unit 230 has a second queue 371 corresponding to the plurality of first queues 351. The queue control unit 230 fetches an entry from one of the plurality of first queues 351, and stores the fetched entry in the second queue 371 corresponding to the plurality of first queues 351. The queue control unit 230 converts the entries stored in the second queue 371 into a plurality of cores 210 corresponding to the second queue 371 (a plurality of cores connected to the queue control unit 230 having the second queue 371). 210). The second queue 371 corresponds to the plurality of cores 210. That is, the number of second queues 371 may be the same as the number of paths between the queue control unit 230 and the first device having a plurality of first queues 351 corresponding to the second queue 371. This is because there is one storage location for entries fetched through one pass. Therefore, if there are more second queues 371 than paths, an extra second queue 371 is generated, and if there are fewer second queues 371 than paths, storing entries in the second queue 371 becomes a bottleneck. In the present embodiment, as illustrated in FIG. 3, there is one path 381 between the queue control unit 230 and the first device, and therefore the second queue 371 for a plurality of first queues 351 included in the first device. Is one. Entries are input / output to / from each of the first queue 351 and the second queue 371 by FIFO (First In First Out), but a method other than FIFO may be adopted as the input / output method.

The queue control unit 230 and the first device may correspond one-to-one, and when the first device includes a plurality of first queues 351, the queue control unit 230 and the first queue 351 group (A plurality of first queues 351) may correspond one-to-one.

A second queue 371 in which a plurality of first queues 351 of the first device (for example, CTL 181) are integrated into one by hardware is provided in the queue control unit 230 of the ASIC 111, and the queue control unit 230 accumulates in the second queue 371. Allotted entries are allocated to a plurality of cores 210.

At that time, as a first comparative example, the number of entries constituting the information unit is specified by looking at the information length written in the header of the information unit (for example, the request length when the information unit is a request), A process of fetching a specified number of entries from the second queue and allocating them to a plurality of cores can be considered. However, in this case, the presence or absence of the header of the information unit is checked for each entry, and as a result, the queue control unit cannot fetch entries from the plurality of first queues 351 continuously. For this reason, the performance impact is great.

As a second comparative example, a process for fetching entries from the first queue to the second queue without considering the information length of the information unit (the number of entries constituting the information unit) can be considered. According to the second comparative example, in order to find the information length written in the header of the information unit, it is not necessary to check the presence or absence of the header of the information unit for each entry. However, in the second queue, there is a possibility that a plurality of entries constituting one information unit are not arranged in succession, that is, the continuity of information units may be lost. Then, it is difficult to transfer entries in units of information units.

In the present embodiment, for each first queue 351, a D index 363 representing the number of consecutive entries constituting the information unit stored in the first queue 351 is provided. The queue control unit 230 fetches entries for the value indicated by the D index 363 from the first queue 351, and stores the fetched entries in the second queue 371. This eliminates the need to check the information length included in the header of the information unit in the first queue 351. Therefore, the ASIC 111 can continuously fetch entries from the plurality of first queues 351. Such queue control is performed by a hardware circuit such as the queue control unit 230 in the ASIC 111. For this reason, it is not necessary for the core 210 in the ASIC 111 to perform queue control. For this reason, an improvement in system performance can be expected.

In this embodiment, the queue control unit 230 continuously fetches and fetches entries corresponding to the value indicated by the D index 363 (all entries included in one information unit) in the second queue. Stored in 371. Meanwhile, the queue control unit 230 does not fetch an entry from the other first queue 351. As a result, in the second queue 371, an entry constituting another information unit is inserted between the first entry and the second entry constituting the information unit, that is, the continuity of the information unit is lost. Can be avoided. As a result, in the case of an extended length information unit that cannot be interpreted unless a plurality of entries are read, some entries are stored discontinuously in the second queue 371 in the ASIC 111, and the multiple extended length information units are correctly stored. You can avoid being unable to interpret the entry. Note that the queue control unit 230 manages from which entry in the second queue 371 to which entry is a continuous entry (information unit). The queue control unit 230 allocates entries from the second queue 371 to the plurality of cores 210 in units stored continuously (that is, information unit units). With this configuration, it is possible to fetch entries while maintaining the continuity of information units without referring to the headers of the information units (in other words, allowing entries to be continuously fetched from a plurality of first queues 351). , Storage and allocation can be performed.

When the number of entries indicated by the D index 363 are successively fetched from the first queue 351 and stored in the second queue 371, the number of fetched entries may be subtracted from the P index 361. The queue control unit 230 that fetched the entry may subtract the number of entries from the P index 361 corresponding to the first queue 351 that is the fetch source, or detects that the C index 353 has a value indicating stored. The first device may subtract the number of entries from the P index 361 corresponding to the C index 353. When both the P index 361 and the D index 363 are indexes updated by the first device, the update of the P index 361 and the D index 363 may always be performed by the first device. At this time, the timing of subtraction of the P index 361 and the update of the D index 363 may be the timing at which the C index 353 is updated (notification to the first device (notification of fetch)).

Further, when the queue control unit 230 stores an entry in the second queue 371 in units of information units, the queue control unit 230 notifies the first device that the entry has been fetched. The value of the corresponding C index 353 is updated to a value indicating that the entry has been stored. When the first device detects the C index 353 that has already been stored (fetched), the first device returns the C index 353 to a predetermined value. The first device can recognize that a new entry can be stored in the first queue 351 corresponding to the C index 353 indicating a predetermined value. For example, when the number of entries remaining in the first queue 351 exceeds a certain number for each first queue 351, the first device stores the C index 353 corresponding to the first queue 351 already stored. An entry may not be added to the first queue 351 unless the value is represented.

Hereinafter, the flow of some processes performed in this embodiment will be described.

FIG. 4 shows a flow of storing information units in the first queue 351.

When the first device stores the information unit (S401: Yes), the information unit is stored in units of entries, and the first queue 351 has a space that is equal to or larger than the total length of the entries constituting the information unit. Store (S402). At this time, the first device adds the number of stored entries to the P index 361 corresponding to the storage destination first queue 351, and stores the number of entries stored in the D index 363 corresponding to the storage destination first queue 351. Record. Note that “when an information unit is stored” means, for example, that there is an information unit to be stored (for example, in a memory in the first device) and at least one first queue 351 that satisfies the condition. If you want, The “first queue that satisfies the condition” may be a first queue that has a free space that is equal to or greater than the total length of the entries that constitute the information unit, and whose corresponding C index 353 indicates a predetermined value.

In S402, if the length of the information unit is equal to or less than the entry length, the first device stores one entry in the first queue 351, adds 1 to the P index 361, and sets 1 to the D index 363. Record. In S402, if the length of the information unit exceeds the entry length, the first device divides the information unit into a plurality of entries, and stores the plurality of entries continuously in the first queue 351, and the P index. The number of stored entries is added to 361, and the number of stored entries is recorded in the D index 363. “Store a plurality of entries continuously in the first queue 351” means that while the plurality of entries are stored in the first queue 351, entries of other information units are stored in the same first queue 351. Do not store. Thereby, the continuity of the entries constituting the information unit in the first queue 351 is maintained. Each entry may be all or a part of the information unit, or may be a predetermined entry length by a combination of all or a part of the information unit and predetermined data (for example, null).

FIG. 5 shows the flow of fetching entries from the first queue 351 to the second queue 371.

When the queue control unit 230 fetches an entry (S501: Yes), it is determined whether or not the D index 363 corresponding to the fetch source first queue 351 is 1 (S502). In the case of “fetching an entry”, at least one P index 361 indicates an integer equal to or greater than 1, and the number of the fetch source is determined by arbitration in at least one first queue 351 corresponding to the at least one P index 361. If one queue 351 is determined, this is sufficient. The first queue 351 that is the fetch source may be, for example, the first queue 351 that stores the largest number of entries (the first queue 351 corresponding to the P index 361 indicating the highest value).

If the D index 363 indicates 1 (S502: Yes), the queue control unit 230 determines that the information unit including the fetch target entry is a standard length information unit (S503). Based on the entry length indicated by the entry length register 373, the queue control unit 230 fetches the number of entries indicated by the D index 363 from the first queue 351 that is the fetch source, and stores the fetched entries in the second queue 371 ( S504).

If the D index 363 indicates any integer greater than or equal to 2 (S502: No), the queue control unit 230 determines that the information unit including the fetch target entry is an extended length information unit (S505). Based on the entry length indicated by the entry length register 373, the queue control unit 230 successively fetches the number of entries indicated by the D index 363 from the first queue 351 that is the fetch source, Store in the queue 371 (S506).

When S504 or S506 is completed (when the entry is stored in the second queue 371), the queue control unit 230 notifies the first device that the entry has been stored from the first queue 351 that is the fetch source. Specifically, the C index 353 corresponding to the first queue 351 that is the fetch source is updated to a value indicating stored (S507). When the first device detects the C index 353 indicating stored (S510), the first device resets the C index 353 to a predetermined value (S511).

In the above flow, the P index 361 corresponding to the first queue 351 of the fetch source may be subtracted by the number of fetched entries.

The queue control unit 230 allocates the entry stored in the second queue 371 to the core 210 selected from the plurality of cores 210 in units of information units, and the processor includes information configured by the allocated entries. A transfer process for transferring the unit to the second device is performed. The “transfer processing” may be, for example, setting the information unit transfer source and transfer destination in the DMAC 220.

By the way, each of the plurality of queue control units 230 includes a base address register 372, an entry length register 373, a depth register 375, a P index 361, and a D index 363, and corresponds to the first device (for server or CTL 181). ing. At least one of the plurality of queue control units 230 is switched to the CTL 181 or the server. This switching is possible, for example, by changing the mode value in the mode register 204.

FIG. 6 shows the mode register 204.

The mode register 204 stores a mode value for each of at least one of the plurality of queue control units 230. The mode value indicates whether it is for the CTL 181 or the server. The queue control unit 230 corresponding to the mode value is executed in the mode (for the CTL 181 or the server) represented by the mode value. In other words, the queue control unit 230 is controlled by the ASIC 111 according to the corresponding mode value. The mode value is switched according to a policy stored in a storage area (for example, a memory) in the ASIC 111 or an instruction from the CTL 181 at the time of environment setting when starting the corresponding first device, for example. Is called.

The CTL 181 is provided with a first queue 351 group (a plurality of first queues 351) corresponding to the processors operating on the CTL 181, but all the first queues 351, such as when a file OS (operating system) operates on the CTL 181, are provided. There may be cases where one queue 351 is not used. On the other hand, in the server 123, the overhead of I / O processing can be reduced by having the first queue 351 group for each core 210 or each virtual OS. For example, NVMe (NVM-Express (Non-Volatile Memory Express)) can support a large number (for example, a maximum of 64,000) of first queues 351, so having as many first queues 351 as possible is expected to improve system performance. Is done. In the present embodiment, when there are first queues 351 that are not used in the CTL 181, these first queues 351 can be used as the first queue 351 in the server 123. For this reason, it can be expected to improve the utilization efficiency of hardware resources, reduce the hardware scale, and improve the system performance.

7 and 8 show the flow of I / O processing. 7 and 8, “SVQ” means a server queue control unit, and “CTQ” means a CTL queue control unit. In the following description, communication between the servers # 0, CTQ # 3, SVQ # 0, and CTL 181 is taken as an example.

The server 123 (hereinafter referred to as server # 0) stores an I / O request (an example of an information unit) in the first queue 351 in the server # 0 in units of entries (S1). At that time, the server # 0 updates the P index 361 and the D index 363 corresponding to the storage destination first queue 351 in the SVQ (hereinafter, SVQ # 3). Further, when the I / O request is a write request, the server # 0 stores the write target data in the memory in the server # 0, and registers the storage location of the write target data in the first SGL. The SGL is a list including a pointer to a memory area in which the divided address information and a pointer to the next address information are stored. The first SGL is stored in the memory in server # 0.

Server # 0 instructs SVQ # 3 to write (S2), and SVQ # 3 notifies one of the cores 210 (for example, core # 0) of activation (S3). Core # 0 instructs SVQ # 3 to read (S4), and SVQ # 3 responds to the instruction by issuing an I / O request (one or more entries) based on P index 361 and D index 363. Fetched from the first queue 351 and stored in the second queue 371 (S5). SVQ # 3 transmits a read completion (I / O request in second queue 371) to core # 0 (S6).

Core # 0 converts the I / O request according to the protocol of the transfer destination CTL 181 and transfers it to the CTQ (hereinafter CTQ # 0) (S8). The CTQ # 0 writes an I / O request to the memory in the CTL 181 and performs I / O activation for the CTL 181 (S9).

The CTL 181 analyzes the I / O request and secures a memory area in the CTL 181 (S10). When the I / O request is a read request, the CTL 181 can read data from the storage device 123123 according to the read request, and store the read data in the memory in the CTL 181. At that time, the CTL 181 registers the data storage location in the second SGL. The second SGL is stored in a memory in the CTL 181.

The CTL 181 generates a DMA request (an example of an information unit) (S11), and stores the DMA request in the first queue 351. At that time, the CTL 181 updates the P index 361 and the D index 363 corresponding to the storage destination first queue 351 in CTQ # 0. The CTL 181 transmits a request acceptance notification to the CTQ # 0 (S12).

CTQ # 0 fetches the DMA request from the first queue 351 (S13), and stores the fetched DMA request in the second queue 371. CTQ # 0 allocates a DMA request from the second queue 371 to the core 210 (for example, core # 0). Core # 0 transmits an SGL read request to SVQ # 3 or CTQ # 0 (S15).

When the I / O request is a write request, the SGL read request is transmitted to SVQ # 3. The SVQ # 3 reads the first SGL from the memory in the server # 0 (S16), and returns a completion notification including the first SGL to the core # 0 (S17).

When the I / O request is a read request, the SGL read request is transmitted to CTQ # 0. The CTQ # 0 reads the second SGL from the memory in the CTL 181 (S18), and returns a completion notification including the second SGL to the core # 0 (S19).

Core # 0 generates parameters (transfer source address and transfer destination address) according to SGL (first or second SGL) in the completion notification (S20), and sets the generated parameters in DMAC 220 (for example, DMAC # 3). DMAC # 3 is activated (S21). The DMAC # 3 performs data transfer between the memory in the server # 0 and the memory in the CTL 181 by DMA according to the set parameters (S22).

DAMC # 3 returns a completion notification to core # 0 after data transfer (S23). The core # 0 transmits a DMA completion notification to the transmission source of the DMA request in S14 (that is, CTQ # 0) (S24). CTQ # 0 stores the response to the request in S12 in the memory in the CTL 181 (S25). The CTL 181 stores an I / O completion notification (an example of an information unit) in the first queue 351, and returns a completion notification to the CTQ # 0 as a response to the I / O activation in S9 (S26). Receiving the notification, CTQ # 0 fetches the I / O completion notification from the first queue 351 and stores it in the second queue 371 (S27). CTQ # 0 allocates an I / O completion notification from the second queue 371 to the core 210 (for example, core # 0) (S28).

Core # 0 that has received the I / O completion notification creates response parameters for each of server # 0 and CTL 181 (S29). The core 210 transmits a response parameter to the SVQ # 3 (S30), and the SVQ # 3 transmits the response parameter to the server # 0 (S31). The core 210 transmits a response parameter to CTQ # 0 (S32), and CTQ # 0 transmits the response parameter to the CTL 181 (S33).

Although one embodiment has been described above, the present invention is not limited to that embodiment.

101: Computer system

Claims (13)

A storage controller for controlling data input / output to / from the storage device;
A server that communicates the storage controller and the information unit according to PCIe;
An intermediary device that mediates communication of information units between the server and the storage controller;
A first device that is one of the server and the storage controller has a plurality of first queues;
The intermediary device has a plurality of processors and a second queue corresponding to the plurality of processors;
First and second indexes are provided for each of the plurality of first queues,
The first device stores an information unit in an entry unit in a first queue selected from the plurality of first queues;
For each of the plurality of first queues, the intermediary device has a base address register indicating the address of the first queue, an entry length register indicating the entry length, and a depth indicating the maximum number of entries that can be stored in the first queue. And a register
Each first index is a value indicating a difference from the pointer indicated by the base address register of the first queue corresponding to the first index,
Each second index is a value indicating the number of entries constituting the information unit stored in the first queue corresponding to the second index,
When the first device stores an information unit in any one of the plurality of first queues, the first device corresponding to the first queue that stores the information unit is based on the number of entries constituting the information unit. Update 1 and 2 index,
For each first queue corresponding to a first index indicating that an entry exists among the plurality of first queues, the mediation device,
If the second index indicates 1, it is determined that the information unit including the fetch target entry is a standard length information unit. If the second index indicates any integer of 2 or more, the fetch target It is determined that the information unit including the entry is an extended length information unit,
Based on the entry length indicated by the length register, the number of entries indicated by the second index is fetched from the first queue, and the fetched entries are stored in the second queue.
When the entry is stored in the second queue, the first device is notified that the entry has been fetched from the first queue;
The intermediary device allocates an entry stored in the second queue to a processor selected from the plurality of processors in units of information units, and the processor assigns an information unit constituted by the allocated entries. , To perform processing for transferring to the second device that is the destination device of the information unit,
Computer system. When the intermediary device determines that the information unit including the entry to be fetched is an extended-length information unit, the number of entries indicated by the second index is continuously obtained from the first queue corresponding to the second index. Do not fetch entries from other first queues while fetching and continuously fetching the number of entries indicated by that second index,
The computer system according to claim 1. Each of the server and the storage controller has a root complex,
The intermediary device is an endpoint connected to the root complex of the server and the root complex of the storage controller,
Each of the server and the storage controller is a device that can be either the first device or the second device;
The intermediary device has a plurality of object sets respectively corresponding to a plurality of first queue groups, each first queue group includes a plurality of first queues, and each object set includes a base address register and a length register. A depth register, a first index and a second index,
Each of the server and the storage controller has one or more first queue groups,
The object set for a server that is an object set corresponding to a first queue group included in the server, and the object set for a storage controller that is an object set corresponding to the first queue group included in the storage controller. including,
The computer system according to claim 2. The mediating device has a mode register;
The mode register stores, for each of at least one of the plurality of object sets, a mode value that is a switchable value indicating whether it is a server object set or a storage controller object set;
The mediating device controls each of at least one of the plurality of object sets according to a mode value corresponding to the object set;
The computer system according to claim 3. Each entry has the same entry length,
The standard length is equal to or shorter than the entry length, and is a length that can accommodate an argument of a frequently used type I / O command,
The extension length is N times the entry length (N is an integer of 2 or more).
The computer system according to claim 4. The first device has a third index for each of the plurality of first queues;
When the intermediary device has the first and second indexes for each of the plurality of first queues and stores an entry in the second queue, the intermediate device corresponds to the first queue from which the entry was fetched. 3 Update the index,
The update of the third index is a notification that the entry has been fetched from the first queue corresponding to the third index.
The computer system according to claim 5. A communication device according to PCIe between devices in a plurality of devices including one or more storage controllers and one or more servers, and mediating the information unit communication,
A server interface that is an interface device connected to the one or more servers;
A storage interface that is an interface device connected to the one or more storage controllers;
Multiple processors,
A queue controller connected to the server interface, the storage interface and the plurality of processors, having a second queue corresponding to the plurality of processors, and controlling the second queue;
A first device that is one of the plurality of devices has a plurality of first queues, and stores information units in entry units in a first queue selected from the plurality of first queues;
An index is provided for each of the plurality of first queues,
Each index is a value indicating the number of entries constituting the information unit stored in the first queue corresponding to the index,
When the first device stores an information unit in one of the plurality of first queues, an index corresponding to the first queue where the information unit is stored is based on the number of entries constituting the information unit. Update
The queue control unit, for each first queue having an entry among the plurality of first queues,
Fetch the number of entries indicated by the index from the first queue, store the fetched entries in the second queue,
The queue control unit allocates an entry stored in the second queue to a processor selected from the plurality of processors in units of information units, and the processor is an information unit configured by the allocated entries. For transferring the information unit to the second device that is the destination device of the information unit,
Mediation device. The queue control unit continuously fetches the number of entries indicated by the index from the first queue corresponding to the index, and continuously fetches the number of entries indicated by the index while the other first Do not fetch entries from the queue,
The mediating device according to claim 7. Each of the one or more servers and each of the one or more storage controllers has a root complex;
The mediation device is an endpoint connected to a root complex of the server and a root complex of the storage controller.
The mediating device according to claim 7. Each of the one or more servers and each of the one or more storage controllers is a device that can be both the first device and the second device;
The one or more queue control units have a plurality of index sets respectively corresponding to a plurality of first queue groups, each first queue group includes a plurality of first queues, and each index set includes a plurality of index sets. Including
Each of the one or more servers and each of the one or more storage controllers has one or more first queue groups,
The server-specific index set in which the plurality of index sets are index sets corresponding to the first queue group of the server; and the storage controller index set that is an index set of the storage controller corresponding to the first queue group; including,
The mediating device according to claim 7. For each of at least one of the plurality of object sets, a mode value storage area that stores a mode value that is a switchable value indicating whether it is a server index set or a storage controller index set;
Each of at least one of the plurality of index sets is controlled according to a mode value corresponding to the index set;
The mediating device according to claim 10. Each entry has the same entry length,
The standard length is not more than the entry length, and is a size that can accommodate an argument of a frequently used type I / O command,
The extension length is N times the entry length (N is an integer of 2 or more).
The mediating device according to claim 7. When the queue control unit stores an entry in the second queue, it notifies the first device having the first queue from which the entry is fetched that the entry has been fetched from the first queue.
The mediating device according to claim 7.

Images