JPH0320781B2 - - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION Technical Field The present invention relates to a proper noun dictionary used for Japanese language information processing in Japanese word processors, office computers, etc., and particularly relates to a data compression method for the Kanji portion of the proper noun dictionary.

BACKGROUND ART Efforts have been made to improve processing efficiency by providing a proper noun dictionary in Japanese information processing devices that use item input methods or kana-kanji conversion methods. The proper noun dictionary is a storage means that stores each proper noun written in Kanji characters having a common attribute with index data and corresponding Kanji data.

In the above item input method, index data is used as an item key, and a proper noun in a kanji character is retrieved by specifying the key in one-to-one correspondence with a proper noun.

On the other hand, in the kana-kanji conversion method, the index data is the reading kana of a proper noun, and the input reading kana is matched with the heading of the kana in the dictionary to extract the proper noun of the kanji.

In either method, conventional proper noun dictionaries require 2 bytes of information to represent one kanji character in the kanji section.

As an example, a kana-kanji conversion method will be specifically explained below. For convenience of explanation, JIS standard C6226 will be used as the kanji code system, and the data representation will be in hexadecimal.

In FIG. 1 showing the concept of a proper noun processing method, 1 is a kana input device, 2 is a processing device, 3 is a proper noun dictionary, 4 is a display device, and 5 is an output device.

When "Toukiyo" is entered on the kana input device 1, the processing device 2 accesses the proper noun dictionary 3,
For example, by sequentially searching for the kana part 3a shown in FIG.
Check the matching with “Tokiyo”. When matching is achieved, the kanji code "Tokyo" in the kanji section 3b is output and sent to the display device 4 or the output device 5 via the processing device 2. The dotted lines with arrows in FIG. 1 indicate the data flow.

The proper noun dictionary 3 in Figure 2 is expressed in a fixed-length format, but in a dictionary designed to reduce memory capacity, a set of kana characters is expressed in a variable-length record format, as shown in Figure 3A, for example. It is common to represent data in kanji. A specific example of one record is shown in FIG. 3B.

The kana code for “Tokiyou” is in the kana part 3aa.
10 bytes and 4 bytes of the kanji codes "456C" and "357E" for "Tokyo" are stored in Kanji section 3bb, but conventionally, even if the data was configured in variable length format, one kanji character was 2 bytes long. Therefore, if you try to construct a proper noun dictionary with thousands or tens of thousands of entries, it is unavoidable that the dictionary's capacity will become enormous.

Purpose of the Invention The purpose of the present invention is to provide a proper noun dictionary in which a set of proper nouns written in kanji is composed of index data expressed in kana codes and corresponding kanji data.
Within the limited free bits of the character code, secure the delimiter information between the original heading part and the kanji part, as well as the delimiter information of the compressed code of the kanji part, and distinguish between the heading part, the kanji part, the kanji code, and the compressed code. The present invention also provides a compression method for a proper noun dictionary that reduces the capacity of the proper noun dictionary.

Summary of the Invention The proper noun dictionary compression method of the present invention provides a proper noun dictionary in which a set of proper nouns written in kanji is composed of index data expressed in kana codes and corresponding kanji data. The frequency of appearance of the kanji that constitutes each proper noun is investigated, and each kanji that appears less than a predetermined frequency is represented by a kanji code with a predetermined data length, and each kanji that appears more than the predetermined frequency is represented by the data length of the kanji code. A file that stores the kanji data expressed in a compressed code having a data length shorter than , and has an identifier attached to the compressed code to distinguish it from the kanji code, and a file in which the compressed code is specified with the compressed code as a headword and a conversion table for converting it into the corresponding kanji code, and set the most significant bit of the kana code of each kana in the index data to either 0 or 1, and each kanji with a frequency of occurrence equal to or higher than the predetermined frequency. is expressed as a 1-byte compressed code, and each kanji with a frequency less than the predetermined appearance frequency is expressed as a 2-byte kanji code consisting of an upper byte and a lower byte, and the most significant bit of the 1-byte compressed data is expressed as 0 or 1. Either the other, and the most significant bit of the upper byte of the 2-byte kanji code is 0 or 1, whichever is the other,
It is characterized in that the most significant bit of the lower byte is either 0 or 1.

EXAMPLES The present invention will be conceived and explained below by way of examples.

FIG. 4 shows a copy of all the data in the kanji section 3b of FIG. 2 onto a magnetic tape 6, which can be run on a computer to check the appearance frequency of each kanji. As a result, when we aggregated the frequency distribution of each kanji character for proper nouns used in names, place names, etc., we found the following unique properties.

(1) It consists of fewer character types than the kanji character types used in independent word dictionaries.

(2) There is a significant bias in the frequency of appearance depending on the type of kanji.

There are two points. The present invention focuses on the characteristic (2) above.

In other words, if we take statistics from a set of several types of proper nouns, we can find the following: field, river, field, mountain, large, island, field, well,
It turns out that the frequency of occurrence of words such as 小, 上, ... etc. is very high, and when we extract, for example, 128 characters in the order of high frequency of occurrence, they are almost 2/3 of the total number of kanji in the proper noun dictionary. I found out that it covers. Furthermore, we found that this coverage rate was always around 2/3, regardless of the number of data items in the proper noun dictionary.

The embodiment compresses the proper noun dictionary by utilizing the properties of these proper noun dictionaries regarding Kanji characters and replacing the usual 2-byte representation of high-frequency Kanji characters with a unique 1-byte code representation.

FIG. 5 shows an example of replacing the JIS/C6226 2-byte code with a 1-byte code. For example, the frequently occurring kanji ``yama'' is expressed as ``3B33'' in JIS, but it is expressed as 1 byte ``03''. "田" is "4544", but 1 byte "00"
Expressed as. Therefore, for the proper noun "Yamada", "3B33 4544" is replaced with "03 00", and the 4-byte value is represented by 2 bytes.

When considering data of 10,000 proper nouns, 1
The number of Kanji characters per data is ~2.5 characters, or ~5 bytes. With 10,000 data, the number of kanji is approximately 25,000.
It is approximately 50,000 bytes. Assuming that 128 high-frequency kanji characters are expressed in 1 byte,
From the above coverage rate, 25000 x 2/3 ≒ 16700 (characters)
will be expressed in 1 byte. i.e. approx.
Memory capacity can be reduced by 16.7K bytes.

The original 2-byte code must be restored from the data compressed in this way, and for this purpose a conversion table 7 as shown in FIG. 6 is provided.

In other words, it is a conversion table that can output the corresponding JIS/C6226 Kanji code when a 1-byte code is set as a relative address. The capacity of this conversion table is 2 bytes per character, so even if there are 128 characters, the capacity is as small as 256 bytes.

In addition, in the kanji part of the compression dictionary, 1-byte and 2-byte representations will be mixed, that is, codes converted to 1-byte and original kanji codes will be mixed, so it is necessary to distinguish between them. There is. In the above example, only the lower 7 bits (128 characters) of one byte are used, so the most significant bit (MSB) is used. This MSB is used as an identifier to distinguish it from original Kanji codes other than high-frequency Kanji characters, and the first byte code is always set to "1". An example of a method for distinguishing from the original Kanji code in relation to this identifier is shown in Figures 7A and B. In Figure A, 8 and 10 indicate 1-byte codes with 1 set to the MSB, and 9 is a 2-byte code sandwiched between them. For the 2-byte code, leave both the MSB of the upper byte and lower byte as "φ" as the original Kanji code. As a result, when reading the kanji part in 1-byte units,
If the MSB is "1", it can be determined that it is a 1-byte code, and if it is "φ", it can be determined that it is a 2-byte code together with the following 1 byte. Next, if the MSB changes to "1", that 1 byte is a 1-byte code. When it is a 1-byte code, when accessing the conversion table 7 in FIG. 6, the MSB is of course inverted to "φ".

The second method is shown in FIG. 7b, in which the MSB of the lower byte of the 2-byte code 9' is left unchanged, and the MSB of the upper byte is set to "1". According to this method, in relation to the kana part 3aa described in FIG. The first part is always "1", and the kana part 3aa
It becomes easy to distinguish the boundary between the kanji part 3bb and the kanji part 3bb.

Note that in FIG. 8, records are shown using only the MSB in 1-byte units. Furthermore, in this example, only the compression method for the kanji part is mentioned and there is no involvement in the compression of the kana part, but as shown in FIG. Therefore, it is also possible to combine various compression methods for the kana part, such as the technique disclosed in Japanese Patent Laid-Open No. 150070/1982, with the compression method of this embodiment.

Although the above embodiment has been explained using the JIS C6226 code system, the present invention is not limited to this code system, and the method according to the embodiment can be applied to other code systems.

In addition, proper noun dictionaries also include personal names and
It is not limited to place names, but various attributes such as company names, trademark names, and product names can be considered. Furthermore, the above compression method can be similarly applied to dictionaries for various computer processing in which the frequency distribution of Kanji characters is biased toward some Kanji characters.

Effects As described in detail above, according to the present invention, in a proper noun dictionary configured with index data representing a set of proper nouns written in kanji with index data expressed in kana codes and corresponding kanji data, Among the kanji in the index data, kanji that are used less frequently are stored in their original 2-byte kanji codes, and frequently used kanji are stored in 1-byte compressed codes. upper bit,
The values of the most significant bits of the upper byte and lower byte of the kanji code of the kanji data and the most significant bit of the compressed code of the kanji data ensure the original delimiter information between the heading part and the kanji part, and the delimiter information of the compressed code. As a result, four types of information with different code lengths can be distinguished using a 1-bit identifier within the limited free bits of the character code, and the proper noun dictionary capacity is effectively used. reduced.

[Brief explanation of the drawing]

Figure 1 is a conceptual diagram of the proper noun processing method using the kana-kanji conversion method, Figure 2 is a schematic diagram of a proper noun dictionary, and Figure 3 shows variable length codes that are the elements of the dictionary. B is an explanatory diagram of an example. Fig. 4 is a schematic diagram of a data file in which only the kanji characters are extracted, and Fig. 5 is an explanatory diagram of an embodiment of the present invention.
Figure 6 is a schematic diagram of a conversion table, Figures 7A and B are explanatory diagrams of the method for identifying converted codes, respectively, and Figure 8 is a variable length record when using the method in Figure 7B. It is an explanatory diagram of an example. 3...proper noun dictionary, 3aa...kana section, 3bb
...Kanji section, 7...Conversion table, 8,10...
1 byte code, 9,9'...2 byte code.

Claims (1)

[Scope of Claims] 1. In a proper noun dictionary composed of a set of proper nouns written in kanji with index data expressed in kana codes and corresponding kanji data, each proper noun in the proper noun dictionary is composed of: The appearance frequency of kanji is checked, each kanji with a predetermined appearance frequency is represented by a kanji code with a predetermined data length, and each kanji with a predetermined appearance frequency or more is compressed with data shorter than the data length of the kanji code. A file that stores the above-mentioned kanji data expressed as a code and with an identifier attached to the above-mentioned compressed code to distinguish it from the above-mentioned kanji code, and a kanji character that corresponds to the above-mentioned compressed code when the above-mentioned compressed code is specified as a headword. The most significant bit of the kana code for each kana in the index data is set to either 0 or 1, and each kanji having a frequency of occurrence equal to or higher than the predetermined frequency is expressed as a 1-byte compressed code. , each kanji with a frequency less than the predetermined appearance frequency is represented by a 2-byte kanji code consisting of an upper byte and a lower byte, and the most significant bit of the 1-byte compressed data is set to either 0 or 1, and the 2-byte The most significant bit of the upper byte of the Kanji code is either 0 or 1, and
A proper noun dictionary compression method characterized in that the most significant bit of the lower byte is either 0 or 1.