JP7654076B2 - Floating-point calculation system, method, and program with threshold prediction - Google Patents

Description

The present invention generally relates to performing floating-point calculations for use in artificial intelligence systems.

Machine learning algorithms, considered as examples of artificial intelligence systems, are becoming widely used in computational applications that benefit from improved iterative computational accuracy. Neural network-based algorithms are one of the most widely used types of machine learning algorithms. A neural network is a model that recognizes underlying relationships in a set of data through a process that mimics the way the human brain operates. A neural network model is first trained using a training data set (training phase), and then the trained neural network model is used to recognize relationships in a target data set (inference phase). While the inference phase may rely on low-precision fixed-point arithmetic, the training phase typically requires floating-point arithmetic.

In one illustrative embodiment, a system includes a floating-point computation unit configured to perform a dot product operation according to a first floating-point value and a second floating-point value, and detection logic operably coupled to the floating-point computation unit. The detection logic is configured to calculate a difference between a fixed-point sum of the exponents of the first floating-point value and the second floating-point value, and detect the presence of a condition prior to completion of the dot product operation by the floating-point computation unit based on the calculated difference. In response to detecting the presence of the condition, the detection logic is further configured to cause the floating-point computation unit to avoid performing a subset of calculations that would normally be performed as part of the dot product operation.

Further illustrative embodiments are provided in the form of an apparatus having a processor and memory configured to execute instruction code, a method configured to perform the detection and avoidance steps, and a non-transitory processor-readable storage medium embodied with executable instruction code that, when executed by a processor, causes the processor to perform the steps, respectively.

Advantageously, as one example, the illustrative embodiments predict a negative dot product output or an acceptably small positive dot product output before the dot product is calculated in order to save early computational overhead on the hardware used to support the computational algorithm.

These and other features and advantages of the embodiments described herein will become more apparent from the accompanying drawings and detailed description that follow.

1 depicts a floating-point number field format by which one or more illustrative embodiments may be implemented. 1 depicts an arithmetic representation of floating-point numbers, with which one or more illustrative embodiments may be implemented. FIG. 1 illustrates a computation kernel associated with floating-point dot product computations by which one or more illustrative embodiments may be implemented. FIG. 1 depicts floating-point multiply-accumulator logic by which one or more illustrative embodiments may be implemented. FIG. 1 illustrates threshold detection logic for floating-point dot product calculations, in accordance with an illustrative embodiment. 1A and 1B depict process flows without and with voltage scaling, in accordance with an illustrative embodiment. FIG. 1 illustrates a method for threshold detection logic for floating-point dot product calculations, in accordance with an illustrative embodiment. FIG. 1 illustrates an exemplary implementation of an artificial intelligence system in accordance with an illustrative embodiment. FIG. 1 depicts an exemplary processor system in accordance with an illustrative embodiment. FIG. 1 illustrates a cloud computing environment in accordance with an illustrative embodiment. FIG. 1 illustrates an abstraction model layer in accordance with an illustrative embodiment.

Illustrative embodiments may be described herein with respect to exemplary computing environments, cloud infrastructures, data repositories, data centers, data processing systems, information processing systems, computer systems, data storage systems, and associated servers, computers, storage units, and devices, as well as other processing and computing devices. However, it should be understood that embodiments of the invention are not limited to use with the particular illustrative system and device configurations shown. Furthermore, terms such as "cloud platform," "cloud computing environment," "cloud infrastructure," "data repository," "data center," "data processing system," "information processing system," "computer system," "data storage system," "computing environment," and the like, herein are intended to be broadly interpreted to encompass other types of systems, including, for example, private and/or public cloud computing or storage systems, as well as distributed virtual infrastructures. However, a given embodiment may more generally include any arrangement of one or more processing devices.

As discussed above in the Background section, the training phase of neural network models used in artificial intelligence (e.g., machine learning) systems typically requires floating-point arithmetic calculations. As an example, such floating-point arithmetic may include calculations defined in the Institute of Electrical and Electronics Engineers (IEEE) standard entitled "IEEE 754-2019: IEEE Standard for Floating-Point Arithmetic". The IEEE 754 standard specifies arithmetic formats and methods for binary (base 2) and decimal (base 10) floating-point arithmetic in a computer programming environment. The IEEE indicates that implementations of floating-point systems conforming to this standard may be realized entirely in software, entirely in hardware, or in any combination of software and hardware.

IEEE 754 defines a half-precision format using 16 bits per number, a single-precision format with 32 bits per number, and a double-precision format with 64 bits per number. Each format includes a sign "s", an exponent "e", and a mantissa expressed as a fractional value "f" (e.g., the number is normalized by assuming one leading bit followed by a fractional part that contains the significant value to the right of the decimal point). Thus, the mantissa is the part of a floating-point number that represents the significant digits of the number, and is multiplied by the base raised to the exponent to give the actual value of the number.

FIG. 1A illustrates a floating-point number field format 100 that conforms to the IEEE 754 standard and that one or more illustrative embodiments may implement. The variables n and p, which indicate the number of exponent and fraction bits, respectively, depend on the selected precision. FIG. 1B illustrates a floating-point number arithmetic representation 110, shown as a value (x), that conforms to the field format 100 of FIG. 1A and that one or more illustrative embodiments may implement.

One of the floating-point calculations performed during the training phase of neural network models used in artificial intelligence systems is the floating-point dot product calculation. The dot product calculation is typically performed in a multiply-accumulator (MAC) unit of an artificial intelligence system. The dot product is an algebraic operation that takes two equal-length number sequences (e.g., two floating-point numbers) as input and returns a single number. More specifically, the dot product is the sum of the products of corresponding entries in the two sequences. If the two input numbers are vectors, the result of the dot product operation (also called the inner product) is a scalar value.

にＲｅｌｕ関数を適用する計算カーネル１２０を描いている。Ｒｅｌｕ関数は、正の入力のみを通す。すなわち、ｙが０よりも大きい限り、Ｒｅｌｕ関数からｙが出力され、そうでない場合は、Ｒｅｌｕ関数は０を出力する。 Furthermore, computational kernels in modern artificial intelligence systems take the inner product (the result of a floating-point dot product operation) and apply a Rectified Linear Unit (Relu) function to the inner product. Figure 1C shows the inner product y computed for two floating-point values x and w.

1 illustrates a computation kernel 120 that applies the Relu function to y. The Relu function only passes positive inputs, i.e., as long as y is greater than 0, the Relu function outputs y; otherwise, the Relu function outputs 0.

の全体を計算する前であっても負の値が検出されて、それによりこの内積を完全に計算することなく０の出力が提供されるようにすることができれば、エネルギー効率が高いであろうことが理解される。例証的な実施形態は、計算アルゴリズムをサポートするのに使用されるハードウェアに関して計算オーバヘッドを早い段階で省くために、内積が計算される前に（すなわち、内積計算が完了する前に）このような負の内積出力を予測する技術を提供する。 In this specification, the dot product

It will be appreciated that it would be more energy efficient if negative values could be detected even before computing the entirety of , thereby providing an output of zero without fully computing the dot product. Illustrative embodiments provide techniques to predict such negative dot product outputs before the dot product is computed (i.e., before the dot product computation is complete) in order to save early computational overhead on the hardware used to support the computation algorithm.

Recall that, as described above and illustrated in Figures 1A and 1B, floating-point numbers are represented with three fields: sign (s), exponent (e), and fraction (f). For the IEEE 754 half-precision format, for example, s = 1 bit, e = 5 bits, and f = 10 bits. Floating-point numbers are represented as value (x) = (-1) ^s * 2 ^[e] * (1.f) for normalized numbers. As is evident, the magnitude of the value depends primarily on the smaller number of bits, "e", and the longer bit stream "f" serves to fine-tune the magnitude of the value. For example, the magnitude of the number x is in the range 2 ^[e] ≦ |x| < 2 ^[e+1] , regardless of the value of "f". It is understood herein that this property advantageously provides significant opportunities for zero prediction in dot-product calculations according to illustrative embodiments.

The product of two floating-point numbers X and Y is restricted as follows:
X=(-1) ^s *2 ^[ex] *(1.fx) => 2 ^[ex] ≦|X|<2 ^[ex+1]
Y=(-1) ^s *2 ^[ey] *(1.fy) => 2 ^[ey] ≦|Y|<2 ^[ey+1]
∴2 ^[ex+ey] ≦|XY|<2 ^[ex+ey+2]

である。したがって、正の積のグループ（Ｓ_ｐ）と負の積のグループ（Ｓ_ｎ）とを区別することができる。 It is further understood herein that, according to an illustrative embodiment, the calculation of the maximum and minimum range of a product does not require multiplications, but only one fixed-point addition of exponent values for two floating-point values X and Y, namely ex+ey. In a dot product calculation, many (N) such products are summed, e.g.

It is therefore possible to distinguish between a group of positive products (S _p ) and a group of negative products (S _n ).

負の積の合計：

The total ranges for the positive and negative groups are as follows:
Sum of positive products:

Sum of negative products:

であることを確認するために、次の条件がチェックされる。

Thus, in the illustrative embodiment, the sum of the products

To ensure that:

Note that the required calculations in equation (1) above involve only fixed-point sums, without expensive multiplications or floating-point calculations. Advantageously, this computationally inexpensive pre-check avoids nearly 50 percent (%) of the calculations (when negative numbers occur). If the number is found to be positive, normal floating-point calculations continue.

As previously mentioned, floating-point dot-product calculations are typically performed in a MAC hardware unit associated with an artificial intelligence system. FIG. 2 illustrates floating-point multiply-add logic associated with a MAC unit 200 by which one or more illustrative embodiments may be implemented. The MAC unit combines three values A, B, and C by adding or subtracting value C to the dot product of values A and B. For floating-point values, the MAC unit 200 processes inputs EA, EB, and EC, where E refers to the exponent portion of each of values A, B, and C. The MAC unit 200 also processes inputs MA, MB, and MC, where M refers to the mantissa (fraction) portion of each of values A, B, and C. The floating-point MAC unit 200 is relatively complex, including alignment logic, adder logic, multiplier logic, shifter logic, comparator logic, and other functional logic (as illustrated in FIG. 2). The conventional logical operation of a multiply-accumulate unit such as floating-point MAC unit 200 will be understood by those skilled in the art and will therefore not be described in further detail herein.

Assuming that fixed-point addition energy is 10% of floating-point based MAC operations, it is understood herein that approximately 40% of MAC calculations can be skipped by predicting negative numbers with 10% prediction overhead. On the other hand, if the accumulated number turns out not to satisfy the pre-check condition, the 10% prediction overhead is wasted. Hence, 0.4*(100% savings-10% overhead)+0.6*(-10% overhead)=30% energy savings.

It is understood herein that in some circumstances, equation (1) may be more conservative than necessary or desirable since the maximum possible magnitude of a positive product is compared to the minimum possible magnitude of a negative product. Thus, in an alternative embodiment, the pre-check condition is relaxed by replacing the "0" in the right-hand term of equation (1) with a threshold value "Th", as depicted below in equation (2).

It should be understood that Th>0 and the magnitude of Th is typically set relatively small. Thus, the calculation can be skipped if the left term is positive but very close to 0. By controlling the Th value, greater energy efficiency is achieved at the expense of acceptably less precision. That is, if an additional 10% of the calculations can be avoided based on the setting of Th, the energy savings is 0.5*(100% savings-10% overhead)+0.5*(-10% overhead)=40% energy savings.

FIG. 3 depicts a logic implementation 300 of threshold detection for floating-point dot product calculations, according to an illustrative embodiment. Logic implementation 300 represents one illustrative embodiment of a hardware implementation of equation (2) above. Alternative embodiments may have other logic implementations.

As shown, logic implementation 300 includes a threshold detector 310 (detection logic) operably coupled to a floating-point MAC unit 340, which is operably coupled to a multiplexer 342. Threshold detector 310 includes logic components including adder 312, adder 314, demultiplexer 316, register 320, register 322, multiplexer 324, exclusive-or (XOR) gate 326, adder 328, and comparator 330. When it is said that a data input, data output, selection bit, control bit (or digital signal), etc. is input or output from a given logic component, this assumes that the given logic component has a corresponding terminal that allows connectivity with other logic components for receiving such corresponding inputs and/or sending such outputs.

であることを想起されたい。したがって、図３のしきい値検出器３１０に示されるように、ＸおよびＹについての指数ビット（指数部）の対応する各対ｅｘ_ｉおよびｅｙ_ｉが加算器３１２に入力され、それにより、加算器３１２の出力はｅｘ_ｉ＋ｅｙ_ｉである。加算器３１２の出力は、マルチプレクサ３２４の出力と共に加算器３１４に入力される。加算器３１４の出力は、デマルチプレクサ３１６に入力される。 As explained above, it is important to note that the calculation of the maximum and minimum range of the product of two floating point values X and Y does not require a multiplication, but only one addition of the exponent values for the two floating point values X and Y, namely ex+ey, and that in a dot product calculation, many (N) such products are summed together, e.g.

3, each corresponding pair of exponent bits for X and Y, ex _i and ey _i, is input to an adder 312, such that the output of adder 312 is ex _i +ey _i . The output of adder 312 is input to adder 314 along with the output of multiplexer 324. The output of adder 314 is input to demultiplexer 316.

を生成し、負の結果を表す、第２の出力上の

を生成する。正の結果はレジスタ３２０に入力され、負の結果はレジスタ３２２に入力される。さらに、正の結果は正レジスタ３２０から出力され、負の結果はレジスタ３２２から出力され、両方はマルチプレクサ３２４に入力され、マルチプレクサ３２４は、前述のように加算器３１４への入力としての働きをする出力を生成する。 The demultiplexer 316 outputs a positive result

on the second output, which represents a negative result.

The positive result is input to register 320 and the negative result is input to register 322. Additionally, the positive result is output from positive register 320 and the negative result is output from register 322, both of which are input to multiplexer 324 which generates an output that serves as an input to adder 314 as previously described.

3, the XOR gate 326 receives as inputs sx _i and sy _i , which are the sign bits for the floating-point values X and Y, respectively. When sx and sy are different, this means that the sign of the product of X and Y is negative. Thus, the value is accumulated with the value in the negative register 322. When sx and sy are the same, this means that the sign of the product of X and Y is positive. Thus, the value is accumulated with the value in the positive register 320. Note that the output of the XOR gate 326 is therefore used to control the demultiplexer 316 and the multiplexer 324, as appropriate.

The positive result from the positive register 320 and the negative result from the register 322 are also input to an adder 328, whose output is checked against a threshold Th in a comparator 330. This is where the pre-check condition of equation (2) is determined. The output of the comparator 330 is provided as an input to EN of the floating-point MAC unit 340 and is also provided as a control signal to a multiplexer 342, which has a first input that is 0 and a second input that is the output of the floating-point MAC unit 340. Note that EN represents an enable switch for the floating-point MAC unit 340 such that the floating-point MAC unit 340 operates only when the output of the comparator 330 is a logical 1 (EN=1). Thus, assuming floating-point MAC unit 340 is configured as shown in floating-point MAC unit 200 of FIG. 2, the input values (i.e., EA, EB, MA, MB, ...) will not be updated in the input registers when EN=0. In this manner, the circuitry of unit 200 of FIG. 2 will be inactive and maintain the previous computation status.

がしきい値Ｔｈ未満である場合は、マルチプレクサ３４２の出力は０であり、そうでない場合は、浮動小数点ＭＡＣユニット３４０の出力がマルチプレクサ３４２の出力として選択される。いくつかの実施形態では、しきい値Ｔｈは０とすることができ、他の実施形態では、しきい値Ｔｈは、比較的小さい何らかの許容可能な正の数とすることができることを想起されたい。例にすぎないが、小さい許容可能な正の数は、ドット積の最大値の約１％よりも小さい値であってよい。 Multiplexer 342 represents the Relu function previously discussed.

If Th is less than the threshold Th, then the output of multiplexer 342 is 0; otherwise, the output of floating-point MAC unit 340 is selected as the output of multiplexer 342. Recall that in some embodiments, threshold Th may be 0, while in other embodiments, threshold Th may be some relatively small, allowable positive number. By way of example only, a small, allowable positive number may be a value less than about 1% of the maximum value of the dot product.

It should be appreciated that in additional embodiments, the circuitry in threshold detector 310 can be implemented with a voltage scaling function such that threshold detector 310 operates below a typical supply voltage to conserve energy. Such voltage scaling function is controlled by voltage scaling controller 344. Controller 344 is configured to enable or disable the voltage scaling function as needed or desired.

Figures 4A and 4B depict a process flow 400 without voltage scaling and a process flow 410 with voltage scaling, respectively, according to an illustrative embodiment. Process flows 400 and 410 correspond to the processing of sequential data (data 1, data 2, data 3, data 4, etc.) associated with threshold detector 310 and floating point MAC unit 340. For example, when floating point MAC unit 340 performs a calculation on a given data (data 1), threshold detector 310 performs a prediction (based on threshold detection) for the next data (data 2), and so on.

Because the prediction stage calculations performed by the threshold detector 310 are relatively simple, the processing delay is relatively small compared to the processing delay associated with the floating point processing of the MAC unit; for example, this relative processing time difference is evident in FIG. 4A. Thus, voltage scaling can be applied to the prediction operation, e.g., the voltage scaling controller 344 causes a percentage reduction in the logic supply voltage VDD (operating or reference voltage) supplied to the circuitry of the threshold detector 310. In some embodiments, the VDD is reduced by 30% of a typical (normal) voltage operating level. The reduced voltage causes the threshold detector 310 to operate proportionately slower, i.e., the processing delay increases. In this case, it is also possible that the time period (time delay) for the floating point calculations is fully utilized by the prediction stage, as depicted in the process flow 410 of FIG. 4B. For example, the processing time for the prediction for data 2 is slowed down to take up all or most of the processing time for the floating point calculations for data 1.

In some embodiments, by reducing the VDD for prediction operations, the overhead is reduced by half (0.7*0.7=0.49, with the formula Energy=C*V _DD ² ), resulting in a 45% energy savings. Such energy savings can be significant, since deep neural networks consist of multiple layers (e.g., up to 50), each layer including convolutions, optional batch normalization, and Relu functions. The scaling of VDD can be applied by controller 344 using any typical voltage control mechanism.

FIG. 5 illustrates a method 500 of threshold detection for floating-point dot-product calculations, according to an illustrative embodiment. It should be appreciated that the method 500 may be performed, for example, by a system including the logic implementation 300 of FIG. 3. However, in alternative embodiments, the method 500 may be performed by other systems configured to perform the steps of the method.

In step 502, the difference between the fixed-point sums of the exponents of the first floating-point value and the second floating-point value is calculated.

In step 504, the presence of the condition is detected based on the calculated difference prior to completion of the dot product operation by a floating-point computation unit configured to perform the dot product operation according to the first floating-point value and the second floating-point value.

In step 506, in response to detecting the existence of the condition, the floating point computation unit is caused to avoid performing a subset of computations that would normally be performed as part of the dot product operation.

6 illustrates an exemplary implementation of an artificial intelligence system 600, according to an illustrative embodiment. As shown, the system 600 includes a training data set 610, a neural network model 620, a threshold detection logic 630, a floating-point MAC unit 640, and a rectification linear unit 650. With respect to the exemplary hardware implementation illustrated in FIG. 3 and described above, it should be appreciated that the adder 312, the adder 314, the demultiplexer 316, the register 320, the register 322, the multiplexer 324, the XOR gate 326, the adder 328, the comparator 330, and the controller 344 can be implemented as part of the threshold detection logic 630. Furthermore, the floating-point MAC unit 340 can be implemented by the floating-point MAC unit 640. Furthermore, the multiplexer 342 can be implemented by the rectification linear unit 650. It should be appreciated that the threshold detection logic 630, the floating point MAC unit 640, and the rectified linear unit 650 are used to perform calculations during training of the neural network model 620 based on the training data set 610.

In an exemplary embodiment, the threshold detection logic 630, the floating point MAC unit 640, and the rectification linear unit 650 of the artificial intelligence system 600 are implemented by one or more application specific integrated circuits (ASICs). An ASIC is an integrated circuit (IC) chip or device that is customized for a specific purpose, comprising logic (e.g., circuits, processors, memory, etc.) that is programmed or otherwise configured with executable program code (e.g., instruction code, computer program code, etc.) for a specific purpose. In this exemplary case, the specific purpose is the implementation and execution of an artificial intelligence system (e.g., a machine learning algorithm), and more specifically, the training phase of the neural network model 620 using the training data set 610. An ASIC may also be considered a system on chip (SoC). Some ASIC implementations that can be used with one or more illustrative embodiments employ cell libraries of user-selectable basic logic functions (e.g., multiplexers and comparators constructed from multiple VLSI transistor devices to provide various functions such as switching and comparison) to enable configuration (and reconfiguration) of the system.

It is further understood that artificial intelligence system 600, and portions thereof, may be realized in alternative circuit/processor-based technologies, such as technologies including one or more multi-core central processing units (CPUs), one or more graphics processing units (GPUs), and one or more field programmable gate arrays (FPGAs). In some embodiments, artificial intelligence system 600 may be implemented as a combination of two or more circuit/processor-based technologies (e.g., ASICs, CPUs, GPUs, FPGAs, etc.).

The techniques depicted in Figures 1-6 may also include providing a system, as described herein, that includes individual software modules, each of which is embodied in a tangible computer-readable recordable storage medium. All of the modules (or any subset thereof) may be on the same medium, or, for example, each module may be on a different medium. The modules may include any or all of the components shown in the figures and/or described herein. In one embodiment of the invention, the modules may run on, for example, a hardware processor. In this case, method steps may be implemented using individual software modules of such a system running on the hardware processor. Additionally, a computer program product may include a tangible computer-readable recordable storage medium having code adapted to be executed to implement at least one method step described herein, including provisioning the individual software modules to the system.

In addition, the techniques depicted in Figures 1-6 may be implemented via a computer program product that may include computer usable program code stored on a computer readable storage medium in a data processing system, the computer usable program code being downloaded over a network from a remote data processing system. Also, in one embodiment of the present invention, the computer program product may include computer usable program code stored on a computer readable storage medium in a server data processing system, the computer usable program code being downloaded over a network to a remote data processing system for use in the computer readable storage medium with the remote system.

An embodiment of the present invention or elements thereof may be implemented in the form of an apparatus comprising a memory and at least one processor coupled to the memory and configured to perform exemplary method steps.

In addition, an embodiment of the present invention may utilize software running on a computer or workstation. With reference to FIG. 7, such an implementation may employ, for example, a processor 702, memory 704, and an input/output interface formed, for example, by a display 706 and a keyboard 708. The term "processor" herein includes any processing device, such as, for example, a multi-core CPU, a GPU, an FPGA, or other forms of processing circuitry, such as one or more ASICs, or a combination thereof. Furthermore, the term "processor" may refer to more than one individual processor. The term "memory" includes memory associated with a processor (e.g., CPU, GPU, FPGA, ASIC, etc.), such as, for example, RAM (random access memory), ROM (read only memory), fixed memory devices (e.g., hard drives), removable memory devices (e.g., diskettes), flash memory, etc. Additionally, the term "input/output interface" as used herein includes, for example, mechanisms for inputting data into the processing unit (e.g., a mouse) and mechanisms for providing results related to the processing unit (e.g., a printer). The processor 702, memory 704, and input/output interfaces such as the display 706 and keyboard 708 may be interconnected, for example, via a bus 710, as part of a data processing unit 712. Suitable interconnections, for example, via the bus 710, may also be provided for a network interface 714, such as a network card, which may be provided for interfacing with a computer network, and for a media interface 716, such as a diskette or CD-ROM drive, which may be provided for interfacing with media 718.

Thus, computer software containing instructions or code for carrying out the methods of the present invention as described herein may be stored in associated memory devices (e.g., ROM, fixed or removable memory) and may be loaded partially or entirely (e.g., into RAM) and implemented by the CPU when ready for use. Such software may include, but is not limited to, firmware, resident software, microcode, etc.

A data processing system suitable for storing and/or executing program code will comprise at least one processor 702 coupled directly or indirectly to memory elements 704 through a system bus 710. The memory elements may include local memory utilized during the actual implementation of the program code, bulk storage, and cache memory that provides temporary storage of at least some of the program code to reduce the number of times the code must be retrieved from bulk storage during implementation.

Input/output or I/O devices (including but not limited to a keyboard 708, a display 706, a pointing device, etc.) may be coupled to the system either directly (such as via a bus 710) or through an intervening I/O controller (omitted for clarity).

Network adapters, such as network interface 714, may be coupled to the system to enable the data processing system to be coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the currently available types of network adapters.

As used herein, including the claims, a "server" includes a physical data processing system (e.g., system 712 shown in FIG. 7) that executes a server program. It will be understood that such a physical server may or may not include a display and keyboard.

The present invention may be a system, method, or computer program product, or a combination thereof, at any possible level of integration of technical detail. The computer program product may include a computer-readable storage medium or media having computer-readable program instructions for causing a processor to implement aspects of the present invention.

A computer-readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridges in grooves on which instructions are recorded, and any suitable combination thereof. Computer-readable storage media in this specification should not be construed as being signals that are inherently ephemeral, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or electrical signals transmitted through wires.

The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to the respective computing/processing device or may be downloaded to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing/processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to be stored in a computer-readable storage medium within the respective computing/processing device.

The computer readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or may be source or object code written in any combination of one or more programming languages, including object oriented programming languages such as Smalltalk® or C++, and procedural programming languages such as the “C” programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or a connection to an external computer may be made (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, including for example programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may utilize state information of the computer readable program instructions to execute computer readable program instructions to personalize the electronic circuitry to implement aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus produce means for implementing the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams. These computer-readable program instructions may also be stored in a computer-readable storage medium capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored constitutes an article of manufacture including instructions that implement aspects of the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device, such that the instructions, which execute on the computer, other programmable apparatus, or other device, implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams, causing the computer, other programmable apparatus, or other device to perform a series of operational steps to create a computer-implemented process.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur in an order other than that noted in the figures. For example, two blocks shown in succession may actually be accomplished as a single step that is performed simultaneously or nearly simultaneously or in a partially or fully overlapping manner, or the blocks may sometimes be performed in reverse order depending on the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or acts or a combination of dedicated hardware and computer instructions.

It should be noted that any of the methods described herein may include the additional step of providing a system with individual software modules embodied in a computer-readable storage medium. The modules may include, for example, any or all of the components detailed herein. In this case, the method steps may be performed using individual software modules and/or sub-modules of such a system running on the hardware processor 702. Additionally, a computer program product may include a computer-readable storage medium having code adapted to be implemented to perform at least one method step described herein, including provisioning the system with the individual software modules.

In any case, it should be understood that the components illustrated herein may be implemented in various forms of hardware, software, or combinations thereof, such as, for example, application specific integrated circuits (ASICs), functional circuits, appropriately programmed digital computers with associated memory, etc. Given the teachings of the invention provided herein, one of ordinary skill in the relevant art will be able to contemplate other implementations of the components of the invention.

Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings presented herein is not limited to a cloud computing environment. Rather, embodiments of the present invention may be implemented in conjunction with any other type of computing environment now known or later developed.

Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with the service provider. The cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

The characteristics are as follows:

On-demand self-service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, on demand, automatically, without the need for human interaction with the service provider.

Pervasive network access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated depending on demand. Consumers generally do not have any control or knowledge over the exact location of the resources provided, although there is a sense of location independence in that they may be able to specify a higher level of abstraction location (e.g., country, state, or datacenter).

Rapid Elasticity: Capacity can be rapidly and elastically, in some cases automatically, provisioned to quickly scale out and released to quickly scale in. To the consumer, the capacity available for provisioning often appears infinite and can be purchased in any amount at any time.

Measured services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services being utilized.

The service model is as follows:

Software as a Service (SaaS): The consumer is offered the ability to use the provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

Platform as a Service (PaaS): The capability offered to the consumer is to deploy applications created or acquired by the consumer, written using programming languages and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, over the application hosting environment configuration.

Infrastructure as a Service (IaaS): The capability offered to the consumer is to provision processing, storage, network, and other underlying computing resources onto which the consumer can deploy and run software of their choice, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, deployed applications, and in some cases, limited control over limited networking components (e.g., host firewalls).

The deployment models are as follows:

Private Cloud: The cloud infrastructure is operated solely for the organization. It may be managed by the organization or a third party and may exist on-premise or off-premise.

Community Cloud: The cloud infrastructure is shared by several organizations to support a specific community with shared concerns (e.g., mission, security requirements, policies, and compliance considerations). It may be managed by the organization or a third party and may reside on-premise or off-premise.

Public cloud: The cloud infrastructure is made available to the general public or large industry groups and is owned by an organization that sells cloud services.

Hybrid Cloud: A cloud infrastructure is a composition of two or more clouds (private, community, or public) that remain unique entities but are held together by standardized or proprietary technologies that allow portability of data and applications (e.g., cloud bursting for load balancing between clouds).

Cloud computing environments are service-oriented and focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

8, an illustrative cloud computing environment 850 is depicted. As shown, the cloud computing environment 850 includes one or more cloud computing nodes 810 with which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or cellular phone 854A, a desktop computer 854B, a laptop computer 854C, or an automobile computer system 854N, or combinations thereof, may communicate. The nodes 810 may communicate with each other. They may be physically or virtually grouped in one or more networks, such as a private, community, public, or hybrid cloud, or combinations thereof, as previously described (not shown). This allows the cloud computing environment 850 to provide infrastructure, platform, and/or software as a service without the cloud consumer having to maintain resources on a local computing device. It is understood that the types of computing devices 854A-N shown in FIG. 8 are intended to be illustrative only, and that the computing nodes 810 and cloud computing environment 850 can communicate with any type of computerized device via any type of network and/or network-addressable connections (e.g., using a web browser).

Referring now to FIG. 9, a set of functional abstraction layers provided by cloud computing environment 850 (FIG. 8) is shown. It should be understood in advance that the components, layers, and functions shown in FIG. 9 are intended to be illustrative only, and embodiments of the present invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

Hardware and software layer 960 includes hardware and software components. Examples of hardware components include mainframes 961, RISC (reduced instruction set computing) architecture-based servers 962, servers 963, blade servers 964, storage devices 965, and networks and networking components 966. In some embodiments, software components include network application server software 967 and database software 968.

The virtualization layer 970 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 971, virtual storage 972, virtual networks including virtual private networks 973, virtual applications and operating systems 974, and virtual clients 975. In one example, the management layer 980 may provide the following functions: Resource provisioning 981 provides dynamic procurement of computing and other resources utilized to perform tasks within the cloud computing environment. Metering and pricing 982 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources.

In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection of data and other resources. User portal 983 provides consumers and system administrators with access to the cloud computing environment. Service level management 984 provides cloud computing resource allocation and management such that required service levels are met. Service level agreement (SLA) planning and achievement 985 provides pre-negotiation and procurement of cloud computing resources where future requirements are anticipated in accordance with SLAs.

The workload layer 990 provides examples of functions for which cloud computing environments are utilized. Examples of workloads and functions provided from this layer include mapping and navigation 991, software development and lifecycle management 992, virtual classroom instructional delivery 993, data analytics processing 994, transaction processing 995, and artificial intelligence algorithms (including threshold detection and floating point calculations) processing 996, in accordance with one or more embodiments of the present invention.

The terminology used herein is merely for the purpose of describing particular embodiments and is not intended to limit the invention. In this specification, the singular forms "a", "an" and "the" are intended to include the plural unless the context clearly indicates otherwise. Furthermore, it will be understood that the terms "comprises" and/or "comprising" as used herein specify the presence of stated features, steps, operations, elements, or components, or combinations thereof, but do not preclude the presence or addition of other features, steps, operations, elements, components, or groups thereof, or combinations thereof.

At least one embodiment of the present invention may provide beneficial effects, such as, for example, a framework (e.g., a set of one or more framework configurations) that replaces complex manual (e.g., custom-built) development of model recovery logic. As illustratively described herein, the framework is configured and instantiated with a set of fault detection components and an associated model recovery pipeline. Once instantiated, the framework plugs into a given lifecycle using logs as input and delivers new model artifacts for the new model version into the existing lifecycle pipeline. In one or more illustrative embodiments, the framework is a cloud-based framework and platform for end-to-end development and lifecycle management of AI applications.

Although the description of various embodiments of the present invention has been presented for illustrative purposes, the description is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been selected to best explain the principles, practical applications, or technical improvements over the art found in the marketplace of the embodiments, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims (18)

1. A system comprising:
a floating-point calculation unit configured to perform a dot product operation according to the first floating-point value and the second floating-point value;
detection logic operably coupled to the floating point calculation unit;
the detection logic is configured to calculate a difference between a fixed-point sum of exponents of the first floating-point value and the second floating-point value and detect the presence of a condition prior to completion of the dot product operation by the floating-point computation unit based on the calculated difference;
responsive to detecting the existence of the condition, the detection logic is further configured to cause the floating point computation unit to avoid performing a subset of computations that would normally be performed as part of the dot product operation. The system of claim 1, wherein the condition detected is whether a result of the calculation of the difference between a fixed-point sum of exponents of the first floating-point value and the second floating-point value is less than a threshold value. The system of claim 2, wherein the threshold value is one of a positive value and 0. The system of claim 2, wherein the existence of the detected condition serves as a predictor that the avoided subset of calculations would result in a dot product of the first floating-point value and the second floating-point value being less than the threshold. The system of any one of claims 1 to 4, further comprising a voltage scaling controller operably coupled to the detection logic and configured to reduce an operating voltage of the detection logic in proportion to a processing delay associated with the floating point calculation unit. The system of any one of claims 1 to 5, wherein the dot product operation is part of a training phase for a neural network model used in an artificial intelligence system. The system of any one of claims 1 to 6, implemented as part of one or more integrated circuits. 1. A method comprising:
calculating a difference between a fixed-point sum of the exponents of the first floating-point value and the second floating-point value;
detecting the existence of a condition based on the calculated difference prior to completion of a dot product operation by a floating-point calculation unit configured to perform a dot product operation according to the first floating-point value and the second floating-point value;
and in response to detecting the existence of the condition, causing the floating-point computation unit to avoid performing a subset of computations that would normally be performed as part of the dot-product operation;
A method wherein one or more of said steps are performed by a processing circuit configured to execute instruction code. The method of claim 8, wherein the condition detected is whether a result of the calculation of the difference between a fixed-point sum of exponents of the first floating-point value and the second floating-point value is less than a threshold value. The method of claim 9, wherein the threshold value is one of a positive value and 0. The method of claim 9, wherein the existence of the detected condition serves as a predictor that the avoided subset of calculations would result in a dot product of the first floating-point value and the second floating-point value being less than the threshold. the difference between a fixed-point sum of the exponents of the first floating-point value and the second floating-point value is

10. The method of claim 9, wherein ex _i represents the exponent of one of the first and second floating-point values, ey _i represents the exponent of the other of the first and second floating-point values, N represents the number of components that make up the exponent, S _p represents a group of positive products, and S _n represents a group of negative products. A method according to any one of claims 8 to 12, wherein the dot product operation is part of a training phase for a neural network model used in an artificial intelligence system. An apparatus comprising:
At least one processor;
at least one memory containing instruction code;
14. Apparatus, wherein said at least one memory and said instruction code, together with said at least one processor, are configured to cause said apparatus to perform the steps of any of claims 8 to 13. A computer program for causing a computer to execute the steps according to any one of claims 8 to 13. 1. A system comprising:
a rectified linear unit;
a floating-point computation unit operatively coupled to the rectified linear unit and configured to perform a dot product operation according to the first floating-point value and the second floating-point value;
detection logic operably coupled to the floating point calculation unit;
the detection logic is configured to detect the presence of a condition prior to completion of the dot product operation by the floating-point computation unit by calculating a difference between a fixed-point sum of exponents of the first floating-point value and the second floating-point value and, based on the calculated difference, comparing the calculated difference to a threshold;
In response to the calculated difference being less than the threshold, the detection logic is further configured to cause the floating point computation unit to avoid performing a subset of computations that would normally be performed as part of the dot product operation and to cause an output of the floating point computation unit to be controlled by the rectifying linear unit. The system of claim 16 implemented as part of an artificial intelligence system. The system of claim 16 implemented as part of one or more integrated circuits.

Images