JP3303045B2 - Network distributed processing system - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a network distributed processing system comprising one or more processor modules interconnected by a communication path between processor modules and a communication network (for example, a WAN), and more particularly to a distributed shared memory between processor modules. The present invention relates to a network distributed processing system capable of efficiently performing message communication by inter-copying.

[0002]

2. Description of the Related Art Conventionally, a distributed processing system in which a plurality of computers perform processing while mutually communicating with other computers has been put to practical use. In addition, a massively parallel system in which a plurality of processor modules are interconnected via an inter-processor communication path can be considered as a kind of distributed processing system because each processor module can be regarded as an independent computer. As a method for performing communication between processor modules in a massively parallel system or between massively parallel systems, for example, a method described in Japanese Patent Application No. 6-202071 and the drawings (Japanese Patent Application No. Hei. There is a multiprocessor system. Hereinafter, this system will be described. FIG. 2 is a configuration diagram of the conventional multiprocessor system. FIG. 2 shows a multiprocessor system 1 including three processor modules 2-1, 2-2, and 2-3. In each of the processor modules 2-1, 2-2, and 2-3, the processors 3-1 to 3-3 that control the entire module, and the corresponding processors 3-1 to 3-2 to 3--3, respectively. 3, local memories 4-1 and 4-2 accessible for reading and writing.
4-3, a distributed shared memory 5-1, 5-2, 5-3 provided with a common address among all processor modules is provided. The processor 3-1 can access only the distributed shared memory 5-1 and cannot access the other distributed shared memories 5-2 and 5-3. Similarly, the distributed shared memory 5-2 can be accessed only from the processor 3-2,
-3 can be accessed only by the processor 3-3.

[0003] These distributed shared memories 5-1, 5-2,
5-3 includes message buffers 6-1 and 6--6, respectively.
4, 6-2, 6-3 and 6-5 are provided. In this case, the message buffers 6-1, 6-2, and 6-3 are the first message buffers in each distributed shared memory, and are assigned the same address. In the processor module, a plurality of message buffers are provided as needed. For example, in the processor module 2-1, the message buffer 6-
4 is provided. In the processor module 2-3, 6-
3, a message buffer 6-5 is provided.
Next, the distributed memory couplers 7-1, 7-2, 7-3
When data is written to the message buffers 6-1, 6-2, and 6-3 of the distributed shared memories 5-1, 5-2, and 5-3, respectively, The written data is transmitted to another processor module (one or more) designated in advance, and the write data received from the other processor module is transmitted to the distributed shared memory 5-1.
5-2, 5-3, the same address location as the transmitting side (that is, the message buffer 6-1, 6-2, 6
-3) is a device for writing. This means that when a write occurs in a certain message buffer 6-1, the write data is sent to, for example, the message buffer 6-2 and copied. Next, the distributed memory protectors 8-1, 8-2, 8-3 respectively control the message buffers 6-1, 6-2, 6-3, 6-4, 6-.
5 is a device for protecting the message stored in No. 5 from unauthorized access. The network adapters 9-1, 9-2, and 9-3 are connected to the communication network 11, and are a kind of input / output device for exchanging messages and the like with other distributed systems via the communication network 11. It is. Note that a wide area network (WAN) or the like can be used as the communication network 11.

The communication path 10 between the processor modules is connected to the processor module 2 in the multiprocessor system 1.
It is a communication medium for transferring messages or their control information between -1, 2-2 and 2-3 at a relatively short distance. On the other hand, the communication network 11 is a communication network for relatively long-distance communication interconnecting the multiprocessor systems 1. The interfaces and protocols of the communication network 11 are generally standardized, and employ a so-called open system architecture structure that allows connection to any system as long as the interface and protocol conditions are satisfied. In such a configuration, the distributed memory couplers 8-1 to 8-3
Stores the destination processor module ID number (PM ID) corresponding to the message buffer address. For example, in order to enable communication from the processor module 2-1 to the processor module 2-2 in the same system, the processor module 2-1 preliminarily stores the message buffer 6-1 address and the received message buffer 6-1 in the distributed memory coupler 7-1. PM ID = ID of PM2-2 ”is stored. As a result, when the transmission PM 2-1 writes a message in the message buffer 6-1 for inter-PM communication, the distributed memory coupler 7-1 causes the reception PM ID (= P
M2-1) is added to the write data and its address, and sent to the processor module communication path 10.
The communication path 10 between the processor modules is connected to the reception PM2-2.
, And the distributed memory coupler 7-2 on the receiving side writes the message to the message buffer 6-2 of the distributed shared memory, thereby realizing communication between PMs in the system.

Next, in the case of inter-system communication for performing message communication between multiprocessor systems connected via the communication network 11, a communication network interface device and communication control software are arranged in each system to standardize. Communication between the systems is performed using the specified protocol. That is, the transmission PM2 in the transmission system
In -1, a message buffer 6-4 for inter-system communication is secured in the distributed shared memory 5-1, control data necessary for controlling the network adapter is set in the network adapter 9-1, and communication protocol processing is executed. . The network adapter 9-1 activates the DMA (Dir) from the message buffer 6-4 by activation from the transmission control program.
The transfer data is read out in the ect memory access mode and transmitted to the communication network 11. Since the transmitting system is the multiprocessor system 1 of FIG. 2, the receiving system is not shown, but it is assumed that the receiving system is the multiprocessor system 1 of FIG. In the receiving system 1, the receiving processor module 2-3 performs communication protocol processing, secures an inter-system communication message buffer 6-5 on the distributed shared memory 5-3, and executes the necessary processing for the network adapter 9-3. Set control data. Since the message buffer 6-5 is captured independently of the message buffer (for example, 6-4) of the transmitting system, the addresses of the two are different. The network adapter 9-3 writes the transfer data received from the communication network 11 to the message buffer 6-5. When the transfer of the message is completed, the network adapter 9-3 interrupts the processor 3-3 to notify the arrival of the message. Finally, the receiving process reads the data from the message buffer 6-5 and performs the corresponding processing.

As another conventional example, a method of performing communication between computer systems in a network distributed processing system in which a plurality of computer systems are connected via a WAN is disclosed in, for example, M.I. T. Tam and D. Farber
'CAPNET-An Approach to Ultra High
Speed Network, Proceedings of the 1990 I
nternational Conference on Communications, pp.
955-961 (1990) '. FIG. 3 is a connection diagram of a network distributed processing system showing the above conventional example. In FIG. 3, three systems 12-1, 12-
2, 12-3 is a wide area network (WAN) 1
3, each system includes distributed shared memories 21-1, 21-2, and 21-3. That is, the three systems 12-1 to 12-3 share a common memory address space (shared space) 21-1 to 21-3, and each of the systems 12-1 to 12-3 stores the original data in the distributed shared memory. Or, a copy thereof (hereinafter, referred to as a cache) is stored. Which node has the original data and which node has the cache is stored in the page table 15 of the exchange 14 in the WAN 13. When performing inter-system communication, the transmitting system (for example, 12-1) writes communication information to a certain address location on the distributed shared memory 21-1. On the other hand, the receiving system (for example, 12
In (3), when reading the data, a memory access request is sent to the exchange 14. The exchange 14 refers to the page table 15 and sends the memory access request to the system (here, 12-1) that requests the ownership of the data. System 12 that received the request
-1 transfers the original data to the requesting system 12-3. Thus, the distributed shared memories 21-1 to 21-1
Communication from the transmission system to the reception system is performed based on the information in the page table 15 in the exchange 14 and 21-3.

[0007]

However, in the former system (the system in FIG. 2), communication within the system can be realized very efficiently by copying between distributed memories, but there are many problems with communication between systems. In other words, in inter-system communication, in order to perform communication using a network adapter, which is a type of input / output device, interrupt processing for processing activation of the input / output device, termination of transfer from the input / output device, or message buffer in the receiving system. It is necessary to execute a lot of communication software, such as capturing of data. Also, by using a standardized communication network interface protocol, it is necessary to execute a number of layered software layers before the data is transferred from the protocol processing transmission process to the reception process, and the overhead and the delay time are increased. There is a problem that it is extremely large. Next, the latter system (the system in FIG. 3) also has some problems. That is, from the viewpoint of the system that sends the memory access request, the memory access request is sent to the system having the ownership, and the original data is returned, so that a large round trip delay occurs. Further, from the viewpoint of a system receiving a memory access request, a software overhead for interrupting a process being executed every time a memory access request is received and performing a data transfer process is large. From the viewpoint of the exchange, the software overhead for keeping the state of the page table up to date is large. Further, from the point of view of the WAN, two message exchanges of a memory request and a page transfer are required for one memory reference, which causes an increase in network traffic. Therefore, an object of the present invention is to solve these conventional problems, reduce software overhead of communication between computer systems via a WAN, reduce transfer delay time to the same extent as communication in a computer system, and increase processing efficiency. An object of the present invention is to provide a network distributed processing system.

[0008]

In order to achieve the above object, in a network distributed processing system according to the present invention, a shared range of a shared memory space is set not only between processor modules in one system but also for a plurality of processors connected to a communication network. To extend the entire system. Therefore, here, 1
A system in which one or more processor modules are logically grouped is called a system. As a connection medium between systems and within a system (between processor modules), a communication path between processors for relatively short distance communication or a relatively long distance communication is used. Any of the WANs may be used. In this way, the message buffer is placed in a shared memory space across the network, with the sending processor module writing to the message buffer and the receiving processor module reading from the message buffer. That is, this is realized by the message passing model. The message buffer on the distributed shared memory physically arranges both the transmitting processor module and the receiving processor module. The transmitting processor module and the receiving processor module each include a distributed memory coupler, and the distributed memory coupler on the transmitting side stores WAN routing information corresponding to the message buffer address on a one-to-one basis or communication path routing information between the processor modules. And an inter-processor communication path interface.

[0009]

According to the present invention, the transmitting processor module selects a message buffer on the distributed shared memory corresponding to each of the destination receiving processor modules, and writes the transfer data therein. The distributed memory coupler extracts WAN routing information corresponding to the write address or communication path routing information between the processor modules, and if it is WAN routing information, a WAN packet in which WAN routing information is added to the memory write address and data is sent to the WAN. On the other hand, if it is the inter-processor module routing information, the local packet in which the memory write address and data are added with the inter-processor module routing information is sent to the inter-processor module communication path. WAN is WAN
The WAN packet is sent to the receiving processor module according to the routing information, and the communication path between the processor modules sends the local packet to the receiving processor module according to the communication path between the processor modules. The distributed memory coupler of the receiving processor module writes data to an address location of the distributed shared memory specified by the WAN packet or the local packet. As a result, both conventional inter-system communication and intra-system communication are similarly performed at high speed and efficiently by copying between distributed shared memories via a WAN or a communication path between processor modules. In other words, a function corresponding to inter-system communication, which conventionally required an extremely large amount of overhead, can be realized with the same light overhead and short delay as in intra-system communication.

[0010]

Embodiments of the present invention will be described below in detail with reference to the drawings. As described above, in the present invention, the connection medium between the systems and within the system (between the processor modules) may be any of a communication path between processors for relatively short-distance communication or a WAN for relatively long-distance communication. . From such an idea, since the method of dividing between system communication and intra-system communication is logically meaningful but not physically meaningful, in this embodiment, communication between processor modules is basically used. Is communication via WAN?
Alternatively, the description will be made separately according to the communication via the communication path between the processor modules. FIG. 4 is a system configuration diagram showing one embodiment of the present invention. In FIG. 4, 17-1
Is a transmission system for transmitting a message, and 17-2 is a reception system for receiving a message. Transmission system 1
7-1 is two processor modules (hereinafter, PM)
18-1 and 18-2, and the receiving system 17-
2 is composed of one PM 18-3. That is,
In this embodiment, not only a system composed of a plurality of PMs but also a system composed of only one PM is included. In FIG. 4, even in the case of one, it is connected to the inter-PM communication path, but the inter-PM communication path is actually unnecessary. Each processor module (PM) has a processor 1
9-1 to 19-3, corresponding processors 19-
Local memories 20-1 to 20-3 that are readable and writable from 1 to 19-3, distributed shared memories 21-1 to 21-2 provided with a common address in all systems and all PMs.
1-3, when data is written in the distributed shared memories 21-1 to 21-3, the write data is transferred to another system designated in advance or another PM in the own system.
And a distributed memory coupler 22-1 for writing write data received from another system or another PM in its own system to the same address location as the transmitting side in the distributed shared memories 21-1 to 21-3.
To 22-3 and distributed memory protectors 23-1 to 23-3
23-3.

The processor 19-1 can access only the distributed shared memory 21-1 and cannot access the distributed shared memories 21-2 and 21-3. Similarly, only the distributed shared memory 21-2 can be accessed from the processor 19-2.
-3 can access only the distributed shared memory 21-3. Further, the distributed memory protectors 23-1 to 23-
3 protects the messages stored in the message buffers in the distributed shared memories 21-1 to 21-3 from unauthorized access. 25-1 and 25-2 are communication paths between PMs, which transfer messages or their control information between PMs arranged at relatively short distances. For example, system bus, multi-stage switching network,
ATM (Asynchronous Transfer Mode) -LAN, ATM switch fabric, etc. are assumed. Of course,
Other types of communication paths between processor modules can also be applied. In addition, since the receiving system 17-2 is configured by one PM, the inter-PM communication path 25-2 may be omitted. Reference numeral 26 denotes a communication network for interconnecting a plurality of PMs arranged at relatively long distances. In this embodiment, transfer data is divided into fixed-length blocks.
An ATM network or a cell relay network in which a 53-byte cell to which a destination header is added and transferred in the network is considered, but other types of communication networks may of course be used.

FIG. 5 is a data layout diagram on the distributed shared memory in FIG. Distributed shared memory 21-1 to 21-
3 at the physical address level,
As shown in FIG. Each distributed shared memory 21-1 to 21-
No. 3 has the same address in the corresponding row, and the same physical address is assigned to each row. The communication area in the distributed shared memory includes (a) a communication area in the system,
It is divided into (b) an inter-system communication area and (c) a FIFO communication area. The intra-system communication area is an area for storing communication information transmitted and received in the same system, and the inter-system communication area is an area for storing communication information between different systems. The FIFO communication area is an area for accumulating and storing, in a FIFO format, a processing request spanning between systems or between PMs in the same system. The intra-system communication area and the inter-system communication area are each divided into the following three areas in units of PM. That is, (i) a message buffer (hereinafter, referred to as MB) area for storing a message, (ii) an MB management map area for displaying a message buffer capture / release state, and (iii) transmission / reception transfer control information such as an MB address. This is a descriptor area for storing (descriptor).

In FIG. 5, distributed shared memories 21-1 to 21-1
In order to identify these areas on 21-3, the identifier of M _ij will be used. Where M is the MB for PM
/ MB management map / descriptor / FIFO communication area identification symbol, MB corresponding to PM18-1 in FIG. 4 is X, its MB management map is XM, and descriptor is XD
It is expressed as Similarly, the MB corresponding to PM18-2 is Y, its MB management map is YM, the descriptor is YD, the FIFO communication area is YF, and the PM18-
The MB corresponding to 3 is named Z, the MB management map is named ZM, the descriptor is named ZD, and the FIFO communication area is named ZF. Here, i and j are the identification numbers (PM I) of the transmitting processor module (PM), respectively.
D) and the identification number (PM ID) of the received PM, and numbers 1, 2, and 3 are assigned to PM 18-1, PM 18-2, and PM 18-3, respectively. However, as an exception, FIFO communication areas XF, YF, ZF
Has only value j and no value i. The reason is FI
This is because the FO communication area is divided corresponding to the receiving PM represented by the value j, and is used in common by the transmitting PMs represented by the value i, so that the value i is not required as area designation information. For example, Z13 is the distributed shared memory 2
M for communication from PM18-1 on PM1-3 to PM18-3
B is represented. XD21 denotes a descriptor from the PM 18-2 on the distributed shared memory 21-1 to the first PM 18-1, and ZF2 denotes a reception processing FIFO communication area to the PM 18-2 on the distributed shared memory 21-3. For simplicity, it is assumed that only one MB, MB management map, and descriptor are prepared for the PM of the receiving system.

As is clear from FIG. 5, when i = j, it indicates a communication area within the same PM. MBs in which the values of ij of the two MBs match and the values of M are different
Form a pair and are located at the same physical address location in different distributed shared memories. For example, in FIG. 5, MBX12 and Y12 form a pair and have the same physical address. Each pair is dynamically captured and released by the kernel of the transmitting PM. Each PM has a fixed length, and an MB management map exists for each PM. For example, in FIG.
Corresponds to the MB management map entry XM12. Here, “correspondence” means that given the head address of the MB (hereinafter simply referred to as MB address),
This means that the MB management map entry address can be obtained, and conversely, the MB address can also be obtained from the MB management map entry address. Each entry in the MB management map indicates whether the corresponding MB is “unused” /
The status of "in use" is displayed, and the value is set by the kernel. Based on the same idea as in the case of the MB, the MB management map entry of the transmission-side distributed shared memory and the MB management map entry of the same address of the reception-side distributed shared memory form a pair. For example, the MB management map entry XM1
2 and YM12 form a pair.

The descriptor area is also allocated to the address location of the distributed shared memory corresponding to the destination reception PM similarly to the MB. An area having the same address as each descriptor on the transmitting side is also arranged in the distributed shared memory of the receiving PM. The descriptor also forms a pair between the same addresses of the transmitting PM and the receiving PM, as in the case of the MB and the MB management map. The correspondence between the descriptor and the MB is dynamically determined by the kernel at the time of execution.
Specifically, for example, when transmitting a message from PM 18-1 to PM 18-2, the kernel of PM 18-1 captures MB (X12), selects the next corresponding descriptor (XD12), and sets the MB address. register. By such an operation, the correspondence between the descriptor and the MB is dynamically determined. The descriptor XD1 on the transmitting side
The descriptor on the receiving side that forms a pair with 2 is YD12
It is. In the FIFO communication area shown in FIG. 5, the distributed shared memory of the transmission PM is composed of a normal RAM,
Processing request FI addressed to PM1 on M distributed shared memory 21-1
The FO area XF1 is a FIFO (First In First)
(Out) memory. For example, the processing request FIFO addressed to PM1 on the distributed shared memory 21-1 of PM18-1
The area XF1 is configured by a FIFO memory,
The processing request FIFO area YF1 addressed to the PM1 on the distributed shared memory 21-2 of the PM 18-2 and the processing request FIFO area ZF1 addressed to the PM1 on the PM18-3 are configured by RAM. Thus, when a processing request is written,
Since these are simultaneously copied to the FIFO communication area XF1 of the PM 18-1 on which the processing request is read, all of these reception processing requests can be stored in the FIFO memory. FIFO of such processing request area
With this, the processing overhead of the processing request between the multiprocessors is automatically reduced by hardware, so that the processing overhead is reduced.

FIG. 1 is an internal block diagram of the processor module shown in FIG. Hereinafter, when it is not necessary to identify the components existing in the three PMs, -1 to -3 are omitted. In FIG.
16 is an MB arranged on the distributed shared memory 21, and 21 is a distributed shared memory. Reference numeral 30 denotes a processor bus for connecting various devices in the PM 18. 19 is a processor and 20 is a local memory. Reference numeral 22 denotes a distributed memory coupler which, when there is a write access to an area mounted on the distributed shared memory 21, also writes to the same address location in another system or the distributed shared memory of another PM in the same system. It has a function of transmitting replacement data and a function of receiving rewrite data from another system or another PM in the same system, and writing the same in the distributed shared memory of the own PM.
The distributed memory coupler 22 includes the following components. First, the bus signal decoder 31 decodes the signal on the signal line of the processor bus 30, and outputs “1” to the signal line 32 when there is a write access to the distributed shared memory to which the bus signal decoder 31 belongs. Start The transfer control unit 33 is a part that controls the entire distributed memory coupler 22, and is supplied with various control signals (not shown) to the internal logic. The routing information management unit 34 stores routing information when data to be written to the distributed shared memory 21 is transmitted / received via a communication network or a communication path between processor modules.

In the transmission PM 18 of FIG. 1, the packet transmission register 35 stores write data to be transferred to another PM in the other system or another PM in the own system, a memory address thereof, and the like. The destination identification decoder 36 identifies whether the transmission packet passes through the PM-to-PM communication path 25 or the communication network 26, and transfers the packet to the local transmission buffer 37 if it is the former, and transfers it to the WAN transmission protocol processor 38 if it is the latter. The WAN transmission protocol processing unit 38 performs protocol conversion and various protocol processing on the transmission packet according to the interface of the communication network 26, and performs WA conversion.
The data is transmitted to the communication network 26 via the N transmission buffer 39. The local reception buffer 40 is connected to the PM communication path 25.
The data transferred from the other PM via the PC is temporarily stored. The WAN reception buffer 41 is connected to the communication network 26.
To temporarily store packets received via The WAN reception protocol processing unit 42 performs various protocol processes according to the interface of the communication network 26, and transfers the received packet to the packet reception register 43. The packet decoder 44 decodes the data in the packet reception register 43, and requests the transfer control unit 33 to execute the writing of the distributed shared memory to the transfer control unit 33 through the signal line 45 when the write request to the shared distributed memory 21 is made. In this way, copying is performed from the transmitting-side distributed shared memory 21 to the receiving-side distributed shared memory 21.

Next, the distributed memory protector 2 shown in FIG.
Reference numeral 3 denotes a hardware mechanism for protecting a message in the MB 16 arranged in the distributed shared memory 21 from unauthorized access. The distributed memory protector 23 has a current capability register (CCR: Current Capability).
Register) 51, MCR memory (Memory Capabilit)
y Register Memory) 50 and a comparator 52. Here, the CCR 51 is a register that stores the current capability. Specifically, the current capability means an ID of an application (APL) object (an object executed in the user mode of the processor) being executed by the processor 19. The MCR memory 50 is composed of a plurality of MCRs 53, and each of the MCRs 53 is
The memory capability is stored for each MB16 above. The memory capability is, specifically, the corresponding M
It means the ID of the application object that has been given the right to access B16. When a memory access to the MB 16 occurs, the corresponding MCR 53 is selected from the MCR memory 50 based on the MB address, and is taken out. The comparator 52 outputs the selected MCR 53 and C
It is determined whether the access to the message buffer is correct by comparing it with the CR 51. If the access does not match, the access is determined to be an unauthorized access. If an unauthorized access is detected, the access to the MB 16 that is being performed in parallel is interrupted, and an emergency notification is sent to the processor via the signal line 54.

A distributed memory protector 23 in a case where the present invention is applied to message communication from a transmission object in the transmission PM 18-1 in the transmission system 17-1 shown in FIG. 4 to a reception object in the reception PM 18-3 of the reception system 17-2. Is as follows. (A) The kernel of the transmission PM 18-1 sets the current capability, that is, the transmission object ID, in the current capability register (CCR) 51-1 before starting execution of the transmission object. (B) The kernel of the transmission PM 18-1 captures the transmission MB 16-1 on the distributed shared memory 21-1 and then transmits the transmission MB 16-1.
The memory capability is set in the MCR 53-1 corresponding to 16-1. The memory capability at this point is MB
Is the ID of the object that has requested capture of the object, that is, the transmission object ID. (C) Each time there is a memory access from the transmission processor 19-1 to the transmission MB 16-1, the distributed memory protector 2
In 3-1, the MCR 53-1 corresponding to the MB address is used.
Is selected and compared with CCR 51-1. M
If the object ID values of CR 53-1 and CCR 51-1 are equal, it is determined that the access is correct, and if they are not equal, it is determined that the access is unauthorized. (D) The message is received PM1 of the receiving system 17-2.
Assume that the receiving MB 16-3 of 8-3 has arrived. The kernel of the receiving PM 18-3 sets the current capability, that is, the receiving object ID, in the CCR 51-3 before starting execution of the receiving object.

(E) The kernel of the receiving PM 18-3 sets the memory capability in the MCR 53-3 corresponding to the receiving MB 16-3. The memory capability at this point is the ID of the object that is the destination of the message, that is, the received object ID. (F) Each time there is a memory access from the receiving processor 19-3 to the receiving MB 16-3, the distributed memory protector 2
In 3-3, the comparator 52-3 compares the MCR 53-3 with the value of the object ID in the current capability register (CCR) 51-3, and determines that the access is correct if they are equal, and that the access is unauthorized if they are not equal. In the above-described operation, during the period when the memory capability and the current capability are set in the MCR 53 and the CCR 51 (protection window release period), only the transmission or reception object to which the ownership of the MB has been given accesses the MB. be able to.
If an unrelated object accesses the MB during the protection window opening period, the MCR
And a CCR capability mismatch, and an unauthorized access is detected. During the period other than the protection window release period, since all application objects including the transmission and reception objects cannot access this MB, very strong memory protection can be achieved.

FIG. 6 is a block diagram showing the internal configuration of the routing information management unit in FIG. As shown in FIG. 6, the routing directory 60 stores a CAM (Content Add).
It comprises a ressable memory (cell) section 61 and a data memory section 62. Here, the CAM is used to input search key data, compare and match the contents of the stored data of each word (CAM cell) all at once, and select and display a CAM cell having a content that matches a specified search condition. Memory. C
Each of the AM cells has “Distributed shared memory page address 6
3 "is stored. Distributed shared memory page address 63
Is a page of an address shared with another system or another PM in the own system, and includes a page address of the MB, a page address of the MB management map,
The page address of the descriptor, the page address of the FIFO communication area, and the like are set. As described in FIG. 5, the MB, the MB management map, the descriptor, and the FIFO
The communication area is allocated to a different page for each PM of the destination system. On the other hand, the data memory 6
Reference numeral 2 denotes a RAM for storing auxiliary data corresponding to the word of each CAM cell. Each entry is composed of an information type field 64 and a routing information field 65.

When the value of the information type field 64 is 0, it indicates that the write data is copied to the copy destination PM via the interprocessor communication path 52. In that case, the routing information field 65 indicates the destination PM.
ID. On the other hand, if the value of the information type field 64 is 1, it indicates that the copy destination PM of the write data is connected to the communication network 26 (ATM network). The routing information field 65 in that case includes VPI (virtual path identifier) information for performing routing of ATM cells. VPI, VCI (Virtual Channel I
(dentifier) is a two-stage identifier for specifying the destination of the cell. Since the values of VPI and VCI are determined when a connection is established between the systems, the values are stored in the distributed memory coupler in advance. For the same distributed shared memory page address 63, the routing information of the receiving PM is registered in the routing directory 60 of the transmitting PM,
By registering the routing information of the transmission PM in the routing directory 60 of the reception PM, even if data is written from the transmission side or the reception side to the area of the distributed shared memory, they are copied to the distributed shared memory of each other, So-called two-way communication can be performed. The MB management map area is an area for performing such two-way communication.

FIG. 7 shows the local / W used in the present invention.
It is a format diagram of an AN communication packet. A local communication packet 80 passing through the PM communication path 25 and a WA passing through the communication network (ATM network) 26
The format of the N communication packet (ATM cell) 89 is
This is as shown at 80 and 89 in FIG. The local communication packet 80 includes a header 81 including a destination PM ID output from the routing information management unit 34, a distributed shared memory address 82, write data 83, and write unit fields 84. The field of the write unit 84 has a write data width, for example, byte, half word (16 bits), long word (32 bits).
Bit) or the like. The WAN communication packet 89 includes a plurality of payload portions (portions other than the header of the local communication packet) of the local communication packet embedded therein using, for example, the ATM AAL5 protocol.
Create a 3-byte fixed length ATM cell. Here, the reason for embedding a plurality of local communication packet payloads in one ATM cell is to increase the transfer efficiency by packing a plurality of local communication packets 80 since the length of the local communication packet 80 is shorter than the ATM cell length. For the packets with the same destination, the operation of combining a plurality of packets into one ATM cell is also performed by the distributed memory coupler 22, but if the required number of packets with the same destination are not collected within a certain period of time, the PA
An ATM cell is created by embedding dummy data called D.

The routing information management section 34 of FIG. 6 operates as follows. That is, when there is a write access to the distributed shared memory 21, the page address and the CAM cell unit 61 are simultaneously compared. When a coincident cell is detected, it means that the address is shared by another system or another PM in the own system, and the corresponding information type 64 and routing information 65 are read from the data memory unit 62. Next, the destination identification decoder 36 shown in FIG. For example, if the information type field 64 is 0, the local communication packet 80 of FIG. 7 is created by incorporating the destination PM ID into the header, and transmitted to the local transmission buffer 37 of FIG. On the other hand, if the information type field 64 is 1, the VPI and VCI are incorporated in the header to create the WAN communication packet 89 in FIG.
The data is sent to the AN transmission protocol processing unit 38.

FIG. 8 is a diagram showing the structure of various data used in the present invention and their reference relationships. First, an object ID 1 uniquely assigned to an application object such as a transmission object or a reception object
70 is the ID of the system in which the object exists
(SYS ID) 171, PM D10 in the system
2. Each field of the local ID 173 in the PM is configured to be uniquely identified in the entire communication network. Next, as shown in FIG. 8, the processing request 174 stored in the FIFO communication area includes the transmission SYS ID 175 of the system that is the transmission source of the processing request,
It is composed of a transmission PM ID 176 of the system and a pointer 177 to the descriptor. Also, the descriptor 180 includes the start address MB of the corresponding MB.
It comprises an MB pointer field 181 for storing A and another message control information field 182. Further, the message buffer MB190 has a pointer NEXTMP191 for connecting a message addressed to the reception object in a list structure, an SID field 192 which is an ID of a transmission object which is a transmission source of the message, and an ID of a reception object which is a destination of the message. RID field 193, a SIZE field 194 indicating the size of the message body (BODY) 196, and an ATT indicating the attribute attached to the message.
It comprises an R field 195 and a BODY field 196 that indicates the content of the message.

For each entry in the MB management map 200, when the value V = 0, the corresponding MB is in the “unused” state, and when V = 1, the corresponding MB is in the “unused” state. Medium ”. When one MB is given, the address of the corresponding MB management map entry 200 is determined by simple calculation from the value of the start address. In the receiving system, FIFO
By reading the communication area periodically, processing request 1
Poll 74. Upon detecting the processing request 174,
The descriptor 180 is read with reference to the descriptor pointer 177, and the address 181 and the message control information 182 of the received MB are known. From the MB address, the address of the corresponding MB management map 200 can also be obtained mechanically.

FIG. 9 is a time chart of the communication between the PMs via the communication network of the present invention, and FIG. 10 is a time chart of the communication between the PMs via the communication path between the PBs. Note that, among the execution steps of the processor in FIGS. 9 and 10, steps indicated by thick ruled lines (for example, FIG.
Step S120 (APL object activation))
Steps executed in the supervisor mode of the processor, and steps indicated by thin ruled lines (for example, step S122 (message writing) in FIG. 9) are executed in the user mode of the processor.

Case 1: P via communication network
Example of Inter-M Communication (See FIG. 9) (a) Processing of Transmission PM In a certain transmission system 17-1, a reception system in which a reception object exists from a transmission PM 18-1 in which a transmission object now exists via the communication network 26. It is assumed that the message is transferred to the receiving PM 18-3 of 17-2. The MB used by the transmitting side is X1 in FIG.
3. The MB management map entry is XM13. Details of the processing contents are described in the descriptor XD13.
The message processing request is written in the processing request FIFO (XF3) for the receiving PM on the transmitting PM 18-1. Further, it is assumed that the execution of the receiving object starts before the execution of the transmitting object ends. The flow of the processing will be described along each step (symbol indicated by SXXX) in FIG. [Step S120] The kernel of the transmission PM 18-1 is
By executing the APL (application) object activation in step S120, the ID (S
ID) at timing t120 to CCR5 of PM18-1.
Set to 1-1 and start execution of the transmission object.

[Step S121] The transmission object requests the kernel to acquire the MB. The kernel checks the destination by referring to the reception object ID, and sends the transmission PM1.
8-1 to the receiving PM 18-3 for communication MB (X1
3) is selected, and the corresponding MB management map entry XM13 is rewritten to "in use" at timing t121 in FIG. At this time, the MB management map entry (ZM13) of the distributed shared memory 21-3 of the receiving PM 18-3 having the same address as that of the XM 13 is also rewritten. This rewriting is performed by the following operation. First, when the processor 19-1 of the transmission PM 18-1 rewrites the MB management map entry (XM13), the routing information management unit 34-1 in the transmission-side distributed memory coupler (see 22-1 in FIG. 1) writes the MB management map entry (XM13). The page address of the management map entry is provided. In this embodiment, for the page address of the XM 13, the information type 64 in FIG. 6 = “1” and the routing information 65 = “VPI indicating destination to reception PM,
Since the "VCI" is stored in the routing information management unit 34-1, they are taken out and transferred to the packet transmission register (see 35-1 in FIG. 1). Next, the destination identification decoder 36 shown in FIG. 1 identifies the communication via the communication network from the information type 64 = '1', and the WAN transmission protocol processing unit 38 shown in FIG. 83 (“in use” display data) Writes a unit of data (specifically, byte write and byte position information) into 53 bytes sandwiching an ATM header 90 and a trailer 92 to create an ATM cell 89, Transfer to the transmission buffer (see 39-1 in FIG. 1). The WAN transmission buffer 39-1 sends the ATM cells to the communication network 26. Communication network 2
6 performs routing by looking at the ATM header 90 and sends it to the distributed memory coupler 22-3 of the receiving PM 18-3 of the receiving system 17-2.

In the receiving-side distributed memory coupler 22-3,
Recognizing that the write is to the shared distributed memory 21-3, the distributed shared memory address (X
The data is written to the ZM13 of the distributed shared memory 21-3 based on the information of the M13), the write data ("in use" display data), and the information of the write unit. In this way, the same value is automatically copied to the destination shared distributed memory by the operation of the distributed memory coupler 22 only by the local side of writing to the shared distributed memory by the writing side of the shared distributed memory. . Therefore, in the present invention, D
A great effect is obtained that does not involve copy overhead by software such as MA. Next, FIG.
At timing t122, the memory capability register (MCR) 53-1 corresponding to the MB (X13) stores the I of the transmission object having acquired the access right to this MB.
D (SID) is registered.

[Step S122] The transmitting object transmits the message to the transmitting side shared memory MB (X1
At 3), writing is performed in units of 4 bytes at timing t123. Every time the 4-byte write occurs twice,
As in the case of "Step S121", the communication network 26 is put together into one ATM cell 89 shown in FIG.
Transmission / reception is performed via the PM, and the distributed memory coupler 22-3 of the receiving PM 18-3 writes twice to the MB (Z13) of the corresponding address of the distributed shared memory 21-3 in 4-byte units. Thereby, the message buffer X
13 to the message buffer Z13. In this message writing process, in a section in which both the values of the CCR 51-1 and the MCR 53-1 are set, the transmission object specified by the MCR and the CCR can access the MB (X13). Other application objects cannot be accessed.

[Step S123] In the transmission object, when the transmission (SEND) processing is requested to the kernel, the kernel clears the MCR (53-1 in FIG. 1) corresponding to the MB (X13) at timing t124. As a result, the protection window opening period ends, and thereafter, it becomes impossible to access the MB (X13) except for the kernel. Next, at timing t125, the transmission kernel registers the control information in the descriptor (XD13) from the transmission PM 18-1 to the reception PM 18-3, and also transmits the processing request 174 in the format of FIG. 8 to the reception PM 18-3. Write to FIFO (XF3). As a result, the distributed memory coupler 22-1 packs the data and addresses to be written to XD13 and XF3 into the ATM cell 89 of FIG. 7 and copies them to the corresponding areas ZD13 and ZF3 of the communication memory 21-3. At this time, the processing request FIFO area (XF3) of the transmission PM 18-1 is configured by a normal RAM, and the processing request FIFO area (ZF3) of the reception PM 18-3 is configured by a FIFO memory. Therefore, the write data to the XF3 is a normal write, but the write to the ZF3 adds new data while keeping the previously written data.
Through the above operation, all information necessary for transmitting the message has been transmitted to the receiving PM 18-3 of the receiving system. [Step 124] The kernel starts the APL object termination processing, and the current capability register CCR
The value of 51-1 is cleared at timing t126 in FIG.

(B) Processing of Reception PM 18-3 [Step S 130] In FIG. 9, the reception system 17
The kernel of the receiving PM at -2 selects the receiving object as the object to be executed next, and the timing t1
The receiving object ID (RID) is received at 30 PM 18
-3 Current Capability Register (CCR) 5
Set to 1-3. [Step S131] When the execution of the reception object is started and the reception object requests the kernel to perform reception processing, the kernel tests whether a message has arrived at the reception object. Here, since the message has not arrived at this point, it is determined that the receiving object should be suspended until the message arrives from the transmitting object. [Step S132] The kernel suspends the receiving object at timing t131.
And current capability register (CCR) 51-3
Is cleared, and information necessary for restarting the reception object is saved in the local memory (see 20-3 in FIG. 1).

[Step S140] At time t140, the kernel of the receiving PM 18-3 periodically checks the processing request FIFO (ZF3) addressed to the own PM to check for a reception processing request. Here, in step t125 of FIG.
Since this request is registered in the O processing request area (ZF3), this request is detected, and the descriptor (ZD13), MB address (address of Z13), and the reception object ID in the MB of the distributed shared memory in the order shown in FIG. The extracted information is set in the local memory (see 20-3 in FIG. 1) necessary for restarting the receiving object. [Step S150] Reception PM of reception system 17-2
The kernel 18-3 selects the receiving object as the object to be executed next, and the timing t1 in FIG.
The receiving object ID (RID) is received at 50 PM 18
-3 Current Capability Register (CCR) 5
Set to 1-3.

[Step S151] When the execution of the receiving object is started and the receiving object requests the kernel to perform a receiving process, the kernel tests whether a message for the receiving object has arrived. At this point, since the message has already been received from the transmission side, the message can be immediately read. Therefore, at timing t151, MCR5 corresponding to MB (Z13)
The receiving object ID (RID) is set in 3-3.
As a result, since the reception object ID (RID) is set in both the CCR 51-3 and the MCR 53-3, the MB (Z13) is protected from being accessed from an APL object other than the reception object. [Step S152] If the receiving object is MB (Z1
The message is read out from 3) at timing t152,
Perform the corresponding processing. MB (Z13) contains a message copied from MB (X13) of the transmission system.

[Step S153] The receiving object requests the kernel to release the MB (Z13). The kernel clears the MCR 53-3 corresponding to MB (Z13) at timing t153. Next, at timing t154
Rewrites the MB management map entry (ZM13) corresponding to MB (Z13) to a value of "empty". The receiving-side distributed memory coupler 22-3 stores in advance the page address of the MB management map and the corresponding routing information (“VPI, VCI indicating the destination to the transmission PM 18-1”). When writing to
The same location (XM1) of the transmission-side distributed shared memory 21-1 via the reception-side distributed memory coupler 22-3, the communication network 26, and the transmission-side distributed memory coupler 22-1.
The value of “empty” is copied to 3). In this embodiment, M
B is always centrally managed on the transmission PM side. FIG.
When the rewriting to the MB management map entry XM13 on the transmitting side is completed at the timing t155, the MB (X13) can be reused on the transmitting PM side at any time after this point. [Step S154] The kernel executes the APL object end processing, and executes the current capability register (C
(CR) 51-3 is cleared at the timing t156 in FIG.

(Case 2) Communication between PMs via the communication path between PMs
In Shin embodiment (see FIG. 10) FIG. 10, since similar to the communication via the communication network of FIG. 9, to be described only the differences between the case 1. First, in case 1, the communication path 26 has passed through the communication network 26.
Via. Therefore, as routing information, V
The difference is that the destination PM ID (see 176 in FIG. 8) is used instead of the PI and VCI. Communication network 2
6 and the interface between the PM communication paths 25 are different in the structure and protocol of the packet to be transferred as shown in FIG. Generally, it is desirable to use an international standard interface in the communication network 26, and in many cases, a protocol that incorporates error control and retransmission procedures is used in order to cope with long-distance communication and communication in a noisy environment. In addition, from the viewpoint of operation cost, the transfer efficiency is often increased by using a protocol that sends as much information as possible in one transfer.

On the other hand, the PM communication path 25 is a local interface inside the system, and the efficiency can be improved by using a simplified and unique interface and protocol. The PM via the communication network of FIG.
In the inter-communication, the transfer is performed by one ATM cell for the 4-byte memory write (maximum) two times to improve the efficiency, whereas the communication via the PM communication path in FIG. For each memory write, a local communication packet (see 80 in FIG. 7) is transferred for simplification. As shown in FIG. 1, the distributed memory coupler 22 itself performs the separation between the local communication and the WAN communication and the transfer processing according to each protocol. Note that, like an ATM, a local LAN communication technology (for example, ATM-LA
N) and WAN (Wide Area Network) communication technology (AT
M public network and cell relay network) are being realized based on the same technology, and both technologies are being integrated . In such a case, the same interface is already used between the systems and within the system, so that a circuit for identifying the two is not required in the distributed memory coupler. Therefore, both the inter-system communication and the intra-system communication are integrated as the same communication, so that the entire system can be simplified.

The present invention is summarized as follows. That is, conventionally, since the communication between the systems was performed via the IO, the overhead for performing the communication between the systems was extremely large. Therefore, in the present invention, based on the fact that communication between systems is also performed via a distributed shared memory,
Share a common distributed shared memory space between systems. Then, the transmission PM of the transmission system and the reception P of the reception system
A distributed shared memory assigned the same address is assigned to M. The transmission PM and the reception PM are WAN or PM
It is connected by an inter-communication path. The distributed memory coupler of the transmission PM determines whether the communication is local communication via the communication path between the PMs or WAN communication via the WAN, and the distributed shared memory on the transmitting side and the distributed shared memory on the transmitting and receiving sides are adapted to the respective interfaces and protocols. By performing copying between the same address locations, a message passing programming model is realized. By using such a method, the inter-system communication and the intra-system communication can be treated as the same inter-PM communication, and the inter-system communication can be realized with the same light overhead as the intra-system communication. If the communication path between the WAN and the PM is set to the same interface, the communication between the systems and the communication within the system can be realized in exactly the same manner except for the difference in physical distance. Therefore, regardless of physical distance, any PM
Can be freely incorporated into one system, and when a PM in a certain system is replaced with another system, it can be performed without any modification.

[0040]

As described above, according to the present invention,
Regardless of the physical arrangement, a distributed processing system over the entire network can be realized simply and extremely efficiently. In particular, when an ATM network is used as a WAN, it becomes easy to add network routing information to transfer information only by hardware using VCI and VPI information of the ATM, so that software overhead is reduced and transfer delay time is reduced. It has a great effect on reduction of the amount.

[Brief description of the drawings]

FIG. 1 is an internal configuration diagram of a processor module according to an embodiment of the present invention.

FIG. 2 is a configuration example diagram of a conventional distributed processing system.

FIG. 3 is a diagram illustrating another configuration example of a conventional distributed processing system.

FIG. 4 is an overall configuration diagram of a distributed processing system according to an embodiment of the present invention.

FIG. 5 is a data layout diagram of a distributed shared memory according to the present invention.

FIG. 6 is a configuration diagram of a routing information management unit in FIG. 1;

FIG. 7 is a format diagram of a local / WAN communication packet used in the present invention.

FIG. 8 is a format diagram of various data in the present invention.

FIG. 9 is a time chart of communication between systems (via a communication network) in the present invention.

FIG. 10 shows a communication (PM) in the system according to the present invention.
6 is a time chart of the inter-communication path.

[Description of Signs] 17-1: Transmission system, 17-2: Reception system, 1
8-1 to 18-3: Processor module (PM), 1
9-1 to 19-3 ... processors, 20-1 to 20-3 ...
Local memory, 21-1 to 21-3 ... distributed shared memory, 22-1 to 22-3 ... distributed memory coupler, 23-
1 to 23-3: distributed memory protector, 25: communication path between PMs, 26: communication network (WAN), 34: routing information management unit, 35: packet transmission register, 36
... destination identification decoder, 37 ... local transmission buffer, 3
8 WAN transmission protocol processing unit, 39 WAN transmission buffer, 40 local reception buffer, 41 WAN reception buffer, 42 WAN reception protocol processing unit, 43
... Packet reception register, 50 ... MCR memory (Memor
y Capability Register Memory), 51: Current Capability Register (CCR), 53: Memory Capability Register (MCR), 60: Routing Directory, 61: CAM (Content Addressable)
(Memory) Cell part, 62 ... data memory part.

Continued on the front page (56) References Hill, M .; D. , Et al, Tempest: A Substrate for Portable Parallel Programs, IEEE COMMON Spring '95 Digest Paper, USA, March 5, 1995, P.M. 327-333 (58) Field surveyed (Int.Cl. ⁷ , DB name) G06F 15/16-15/177 G06F 9/46

Claims (3)

(57) [Claims] 1. A network distributed processing system in which one or more processor modules included in each processing system are interconnected via both a communication between processor modules and a communication network, wherein each of the processor modules has a memory space. A shared memory sharing the same, and the communication corresponding to the address of the distributed shared memory on a one-to-one basis.
Routing information Hoa Rui Shin network stores the routing information of the processor <br/> inter sub-module communication path, 該記
Using the stored routing information, the processor module
Via any of the communication paths between
And even communicate with, and a distributed memory control means for copying the <br/> write data between the same address location of the distributed shared memory of each processor module, the dispersion of the processor module of the transmitting side When data is written to the shared memory, the processor module on the transmission side
The said distributed memory control means, routing information Hoa Rui said communication network corresponding to the write address takes the routing information of the inter-processor module communication path, when retrieving the routing information of said communication network, said routing information sends a first packet appends to the memory write address and data to the processor module of the receiving side via the communication network <br/> click, before
The routing information of the communication path between the processor modules.
Ri when issued, said routing information memory write address and a second packet to the receiving side via the communication path between the processor module processor appends to the data
Sent to the module, distributed memory control means of the processor module of the receiving side, the first packet store Rui copying the data to the same address location of the distributed shared memory designated by said second packet Characteristic network distributed processing system. 2. The network distributed processing system according to claim 1, wherein a structure of said second packet is changed to said second packet .
A network distributed processing system characterized in that the structures of both packets are matched so that they can be incorporated into one packet, and the interface of the communication path between the processor modules is the same as the interface of the communication network. 3. The method according to claim 1 or 2 ,
In the network distributed processing system according to either of the communication network and the ATM network, as routing information of the communication network, the A
A network distributed processing system using a virtual channel identifier and a virtual path identifier whose values are previously determined in a TM network .