Exponential Renyi’s entropy of ‘Type (a, ß)’ and new mean code-Word length

Shannon's entropy [13] , is defined as 1 ( ) log

Various generalized entropies have been introduced in the literature, taking the Shannon entropy as basic and have found applications in various disciplines such as economics, statistics, information processing and computing etc.

Generalizations of Shannon's entropy started with Renyi's entropy [12] of order- , given by =1 1 ( ) = log , 1; 0 1

Campbell [3] studied exponentials of the Shannon's and Renyi's entropies, given by represent respectively the Shannon's and Ranyi's entropies. It may also be mentioned that Koski and Persson [7] studied ( , ) () ( , ) ( ) ,

exponential of Kapur's entropy [6] given by =1 ( , ) =1 1 ( ) log , ; , 0 ()

It is interesting to notice that, in the case of discrete uniform distribution , n A (1.3), (1.4) and (1.5) all reduce to n , just the 'size of the sample space of the distribution'.

This paper is organized as follows: Sec.II, define the exponential entropy of 'type ( , )

' and discuss its some major properties corresponding to exponential entropy of concave function. Sec. III, discuss a measure of length and discuss the relationship between the and study its properties.

Exponential Entropy of 'Type ( , )

Corresponding to Renyi's entropy of 'type ( , )  ', the exponential entropy of 'type ( , )  ' is defined as follows:

 ' entropy of a discrete distribution A is given by:   (2.1) becomes exponential entropy of type  or  .

The quantity (2.1) introduced in the present section is entropy. Such a name will be justified, if it shares some major properties with Shannon's and other entropies in the literature. We study some such properties in the next theorem. (0,1)

To prove the next property, we shall use the following definition of a concave function.

Definition :( Concave Function

Similarly, for 1; 1   (2.4) holds. Therefore () EA   is a concave function for all , 0; .

    

In the next section, we will define a new exponential mean length and prove a lower bound () EA   .

A Measure of Length

In the ususal discussion of the coding theorem for a noiseless channel (Feinstein [4] ) one choose code lengths to minimize the average code length. The minimization is done subject to the constraint that the code be uniquely decipherable. The solution of this minimization problem is that the best code length for an input symbol of probability p is log p 

. This solution has the disadvantage that the code length is very great if the probability of the symbol is very small. Implicit in the use of average code length as a criterion of performance is the assumption that cost varies linearly with code length. This is not always the case. In the present paper another measure of code length is introduced which implies that the cost is an exponential function of code length. Linear dependence is a limiting case of this measure. A coding theorem analogus to the ordinary coding theorem for a noiseless channel will be proved. The theorem states that it is possible to encode so that the measure of length is close to the exponential Renyi's entropy of 'type ( , ) ' of the input.

Let a finite set of n input symbols

x  be encoded using alphabet of D sysmbols, then it has been shown by Feinstien [4] that there is a uniquely decipherable code with lengths 12 , ,..., n N N N if and only if the Kraft inequality holds, that is,

where D is the size of code alphabet. Furthermore, if

is the average codeword length, then for a code satisfying (3.1), the inequality

3) is also fulfilled and equality holds if and only if log ( ) ; 1, 2,...,

and that by suitable encoded into words of long sequences, the average length can be made arbitrarily close to () HA, (see Feinstein [4] ). This is Shannon's noiseless coding theorem. By considering Renyi's entropy (see e.g. [12] ), a coding theorem and analogous to the above noiseless coding theorem has been established by Campbell [2] and the authors obtained bounds for it in terms of () HA  . Kieffer [8] defined a class rules and showed () HA  is the best decision rule for deciding which of the two sources can be coded expected cost of sequences of length N when N , where the cost of encoding a sequence is assumed to be a function of length only. Futher, in Jelinek [11] it is shown that coding with respect to Campbell's mean length is useful in minimizing the problem of buffer overflow which occurs when the source symbol is produced at a fixed rate and the code words are stored temporarily in a finite buffer. There are many different codes whose lengths satisfy the constraint (3.1). To compare different codes and pick out an optimum code it is customary to examine the mean length, 1 n ii i Na   , and to minimize this quantity. This is a good procedure if the cost of using a sequence of length i N is directly proportional to i N . However, there may be occasions when the cost is more nearly an exponential function of i N . This could be the case, for example, if the cost of encoding and decoding equipment were an important factor. Thus, in some circumstances, it might be more appropriate to choose a code which minimizes the quantity

, where ,  is some parameter related to the cost. For reasons which will become evident later we prefer to minimize a monotonic function of C . Clearly this will also minimize C . In order to make the result of this paper more directly comparable with the usual coding theorem we introduce a quanity which resembles the mean length. Let a code length of order ,  be defined by , Remark: 2. Huffman [5] introduced a procedure for designing a variable length source code which achieves performance close to

References

1. Arndt C. Information measure-information and its description in science and engineering, Springer, Berlin. 2001.
2. Campbell LL. A coding theorem and Renyi's entropy, Inform. Control, 1965; 8:423-429.
3. Campbell LL. Exponential entropy as a measure of extent of distribution, Z. Wahrscheinlichkeitstheorie verw. Geb, 1966; 5:217-225.
4. Feinstein A, Foundation of Information Theory, McGraw Hill, New York, 1956.
5. Huffman DA. A method for the construction of minimum redundancy codes, in: Proceedings of the IRE. 1952; 40:1098-1101.
6. Kapur JN. Measure of information and their applications,1st ed., New Delhi, Wiley Eastern Limited, 1994.
7. Koski T, Persson LE. Some properties of generalized exponential entropies with applications to data compression, Information Sciences, 1992; 62:103-132.
8. Kieffer JC. Variable Lengths Source Coding with a Cost Depending Only on the Codeword Length, Information and Control, 1979; 41:136-146. doi: 10.1016/S0019-9958(79)90521-7.
9. Kumar S, Choudhary A. Some coding theorems based on three types of the exponential form of cost functions, Open Systems & Information Dynamics, 2012; 19(4).
10. Kumar S, Kumar R, Choudhary, A., Some more results on a generalized parametric R-Norm information measure of type  , Journal of Applied Science and Engineering, 2014; 17(4):447-453.
11. Jelinke F. Buffer overflow in variable length coding of fixed rate sources, IEEE Transactions on Information Theory, 1980; 3:490-501.
12. Renyi A. On measures of entropy and information, Proc. 4th Barkley Symp. on Math. Stat. and Probability, University of Califorma Press, 1961; 1:547-561.
13. Shannon CE. A mathematical theory of communication, Bell System Tech. J. 1948; 27:378-423, 623-656.
14. Garg D, Kumar S. Parametric R-norm directed-divergence convex function, Infinite Dimensional Analysis, Quantum Probability and Related Topics, World Science Journals, 2016; 19(2). DOI: 10.1142/S0219025716500144
15. Garg D, Kumar S. An application of generalized Tsalli's-Havrda-Charvat entropy in coding theory through a generalization of Kraft inequality, International Journal of Applied Research, 2016; 1(4):01-05, ISSN: 2394-5869.
16. Garg D, Kumar S. Some Remarks On 'Useful' Renyi's Entropy of Order α in International Journal of trends in Mathematics and Statistics (IJTIMS), 2015; 4(10):178-183. ISSN: 2279-0318.