Uncertainty estimation of mutual information with joint distribution of marginal random variables

Introduction

In recent times, mutual information has become a crucial indicator of statistical reliance. In engineering, particularly in machine learning, it has been applied as a criterion for feature selection (see [1][2][3] and the references therein). Shannon originally presented its antecedent, known as the rate of transmission, in a classic work [4] for the communication system in 1948. Entropy was initially established by Shannon for a single discrete chance variable. Using the joint distribution, he subsequently established the joint entropy and conditional entropy for two discrete chance variables. Lastly, he stated that the difference between the entropy and conditional entropy represents the rate of transmission. Different definitions of mutual information have been proposed since Shannon's groundbreaking work [4] . These definitions can be broadly classified into two classes: (1) definitions involving random variables and (2) definitions involving ensembles, or in mathematical literature, probability spaces. The joint distribution of two random variables is necessary for the definition of mutual information. More specifically, Kullback (1959) [5] defined entropy, conditional entropy, and joint entropy using elementary mathematical techniques. Using probability theory, Pinsker (1960, and 1964]) [6] tackled Shannon's basic [15,16] ideas in a more sophisticated way. His definition of mutual information was more expansive, implying that the two random variables had different probability domains. Using Shannon's method, Ash (1965) [7] defined mutual information, specifically presuming that the two random variables shared the same probability space [17] . The joint probability measure of the joint sample space of two ensembles determines the definitions. Three authors of such definitions-Fano (1961) [9] , Abramson (1963) [10] , and Gallager (1968) [11] -evolved their concept in a similar way. They defined condition entropy, ensemble entropy, and joint entropy of two ensembles first. They then clarified what a joint event's mutual information was. They determined the mean value of the mutual information of a joint event, noting that it is a random variable. They dubbed the outcome the mean information of two ensembles.The definition of y's conditional entropy 𝐻 𝑥 (𝑦) is

The difference between H(x) and 𝐻 𝑦 (x) is the definition of the information transmission rate, or R. As a result, there are two methods to express R:

Remark 11.Shannon did not discover the precise equation for R: R = ∑ p(i, j) log p(i,j) p(i)p(j) i,j (1.4) Give the following definitions for the joint, entropy, and conditional entropies:

Then H(x, y) = H(x) + H(y/x) ≤ H(x) + H(y) and H(y) ≥ H(y/x) Ash [7] started with two arbitrary variables. X and Y made the assumption that their probability spaces were identical. By using Shannon's approach, he methodically described the entropy, conditional entropy, and joint entropy in [4] . Finally, he indicated H(X) -H(X/Y) by I(X/Y) and referred to it as the data that Y had sent regarding X. Pinsker [6] used probability theory to address Shannon's basic ideas in a more sophisticated way. Assume that there are two random variables δ and μ: one defined on a probability space (Ω, 𝑆 𝜔, ,𝑃 𝛿 ), and the other one defined on a probability space (Ψ, 𝑆 𝜑, 𝑃 𝜇 ), and both of them are taking values in a measurable space (X, 𝑆 𝑥 ) and ((Y,𝑆 𝑦 ). Then, the pairδ, μ of random variables may be regarded as a single random variable (δ, μ) with values in the product space X×Yof all pairs (x, y) with x∈X, y∈Y. The distribution of (,) is referred to as the joint distribution of random variables δ and μ.The product of the distributions𝑃 𝛿 (. )and 𝑃 𝜇 (. ) represents the distribution 𝑃 𝛿𝑥𝜇 (. )defined on 𝑆 𝑥 × 𝑆 𝑦 .

𝑃 𝛿𝑥𝜇 (𝐸𝑋𝐹) =. 𝑃 𝛿 (𝐸)x𝑃 𝜇 (𝐹) (1.6) For E ϵS x and F ϵS y If X and Y are discrete random variables with countably many points each, for example,{x 1 , x 2 , . . . . . }..and{y 1 , y 2 , . . . . . } then

(1.7) I(δ, μ) is called the information of δ and μ with respect to the other. Assume X is a discrete variable with the probability mass function P(x) = Pr{X= x}, x∈X and the alphabet X. The probability mass function p(y) = Pr{Y = y},, y∈Υ and the discrete variable Y with the alphabet Y are both assumed. Suppose that the joint mass function (joint distribution) of X and Y is p(x, y). The mutual information I(X,Y) can then be defined as

Gallager [11] later introduced the idea of mutual information into communication theory using a more complete and in-depth method. In fact, he merged and combined the findings from Abramson [10] and Fano [9] . Let's assume that each discrete ensemble, X and Y, has a sample space {a 1, a 2, . . . . . . . . a k } and {b 1, b 2, . . . . . . . . b J } Consider the joint sample space {(𝑎 𝑘, ,𝑏 𝑗, ):1≤ k ≤ K, 1 ≤ j ≤ J}. A probability measure on the joint sample space is given by the join probability 𝑃 𝑋𝑌 (𝑎 𝑘, ,𝑏 𝑗, ), defined for 1≤ k ≤ K, 1 ≤ j ≤ J.

It is thus possible to determine the marginal probability as

𝑃 𝑋𝑌 (𝑎 𝑘, , 𝑏 𝑗, ), j = 1,2, . . . . . . . . J 𝐾 𝑘=1

(1.9) This is expressed in shorter notation as

Given that outcome 𝑎 𝑘 of x is, the conditional probability that outcome y is 𝑏 𝑗, if 𝑃 𝑋 (𝑎 𝑘 )>0,>0 is defined as = 𝐼 𝑌:𝑋 (𝑎 𝑘, , 𝑏 𝑗, )

By means of an information channel consisting of a transmitter of alphabet A with elements 𝑎 𝑖 and total elements t and a receiver of alphabet B with elements 𝑏 𝑗, and total elements r, Abramson [10] A,B by I(A,B) and called it mutual information of A and B. The rate of transmission in [1] or the mutual information I(X,Y) between two continuous random variables X and Y [8] is defined as I(X, Y) = ∬ P(x, y) log P(x,y) P(x)P(y) dx dy, (

However, differential mutual information is much less prevalent than discrete mutual information. On the one hand, the joint density function in question needs to be calculated because it is typically unknown [13,14] . However, as most data utilised in machine learning and engineering are finite, mutual information between discrete random variables is used.

Our Claims

In information theory Kullback's formulation was more concise and mathematical. While Ash's [12] definition adhered to Shannon's approach, it was more methodical. Because Pinsker's formulation made use of probability theory, it was the most mathematical. In terms of communication theory, Gallager's definition was more thorough and comprehensive. Because of its brevity, Cover and Thomas's concept has become accepted as standard practice in information theory. Some mathematical errors exist in these different definitions of mutual information, though. Marginal probabilities are redefined from the joint probabilities. In actuality, the marginal probabilities need not be redefined from the joint probabilities since they are provided by the ensembles. With the exception of Pinsker's definition, either assume that the two random variables have the same probability space or ignore the probability spaces. The above definitions make the assumption that there is a joint probability measure or distribution. However, they all fail to acknowledge a crucial fact: the joint probability measure or joint distribution is not unique. Now, if X be a finite discrete random variable on discrete probability space (Ω

The joint distribution becomes more complex, though, when X and Y have separate probability spaces and, consequently, different probability measures.

A joint probability measure of 𝑃 𝑋𝑌 of 𝑃 1 and 𝑃 2 is

Claim 2.1. The product probability measure, also known as the product distribution, is the joint probability measure that exists for any two discrete random variables.

Proof

Similarly, 𝑃 𝑋𝑌 (Ω 1 /𝐵) = 𝑃 2 (𝐵) for any B ∈ Ω 2 . Hence, 𝑃 𝑋𝑌 ({𝑋 = 𝑥 𝑖 }/{𝑌 = 𝑦 𝑗 })=𝑃 1 ({𝑋 = 𝑥 𝑖 }𝑜𝑟{𝑌 = 𝑦 𝑗 }) is a combined X and Y probability measure (see) [8] . We study the discrete probability distributions P and Q. Then D(P // Q) ≥ 0 with equality if and only if 𝑝 𝑖 = 𝑞 𝑖 for all i. Since the Kullback-Leibler distance is not symmetric and does not meet the triangle inequality, it cannot be considered a real distance between distributions. However, it is frequently helpful to consider relative entropy to be a "distance" between distributions. The subsequent characteristic demonstrates that the Kullback-Leibler distance between the joint distribution 𝑃 𝑋𝑌 and the product distribution 𝑃 𝑋 𝑃 𝑌 represents mutual information under a joint probability measure.

Claim 2.2:

The Kullback-Leibler distance between the joint distribution and the product distribution is the mutual information of random variables X and Y.

Proof. Through the use of a mapping from two-dimensional to one-dimensional indices, (i, j) → (i -1) * L + j ≜ 𝑛 for i = 1, . . . . . . . , K, j = 1,2, . . . . . . . , L, if and only if X and Y are independent, then with equality.

Proof. To demonstrate the claim 2.2, let's utilise the mappings between two-dimensional indices and one-dimensional indices.

I(X, Y) ≥ 0 with equality if and only if P XY (x ⌈n/L,⌉ y n-(⌈n/L⌉-1) * L )=𝑃 1 (𝑋 = 𝑥 ⌈𝑛/𝐿⌉ )𝑃 2 (𝑌 = 𝑦 n-(⌈n/L⌉-1) * L ) for n = 1, 2,..., L, L+ 1,..., 2L..., (K -1)L +...+K that is, 𝑃 𝑋𝑌 (𝑥 𝑖, , 𝑦 𝑗, )=𝑃 1 (𝑋 = 𝑥 𝑖 )𝑃 2 (𝑌 = 𝑦 𝑗 ) for i = 1,..., K and j = 1, 2,..., L, or X and Yare are separate.

Proof. Suppose the range of X is a constant x and the sample space has only one point ω Then,𝑃 1 (𝑋 = 𝑥) = 𝑃 1 (𝜔) = 1.

Thus, X and Y are independent.

Claim 2.4 Let X be discrete random variables with K values. Then 0 ≤ H(X) ≤ log K (2.12)

Proof. Consider two discrete random variables, X and Y. Then, the connections between conditional entry, entropy, and mutual information are as follows: Proof. If every value of X is unique, then K=N, or the number of distinct values of X equals the number of observations. we observe that (1) For all i = 1, 2,..., K and j = 1, 2,..., L, 𝑂 𝑖𝑗 = 0 or 1.

(2) For all i = 1, 2,..., K and j = 1, 2,..., L, 𝑝 𝑖𝑗 = 𝑂 𝑖𝑗 /N = 0 or 1/N

(3) for each j = 1, 2,..., L since 𝑂 1𝑗 +𝑂 2𝑗 +………𝑂 𝐾𝑗 =𝑛 𝑗 Nonzero's 𝑂 𝑖𝑗 or similar 𝑛 𝑗 nonzero 𝑝 𝑖𝑗 's, are present.

(4) 𝑝 𝑖 ⋅ = 1/N, i = 1, 2,..., K.

With the aforementioned findings and the knowledge that 0 log 0=0, we have

Claim 2.6 Let p and q be two positive numbers less than or equal to 1. Then

The equality holds if and only if p=q. Additionally, p is near to q.

Proof. Let's now use 𝐼 ̂ (X, Y) ̂ to represent N× I(X, Y). Then, applying claim 2.5, we obtain 2𝐼 [5] states that it has a degree of freedom of (K -1) (L-1) and follows a distribution. Therefore, 2N × I(X, Y) roughly follows a ℵ 2 distribution with (K -1) (L -1) as its degree of freedom. The well-known Chi-square test for determining whether two random variables are independent is this one. This enables the assignment of a significant level that corresponds to the values of mutual information and (K-1) (L-1) using the Chi-square distribution.

These are the null and alternative hypotheses

𝐻 0: There is no tie between X and Y; they are independent. 𝐻 1 :There is a relationship between X and Y, meaning that they are dependent on one other. The null hypothesis is rejected at the α significance level according to the decision rule if the ℵ (2.21)

Conclusions

Using random variables with distinct probability spaces, we have provided a unified definition of mutual information in this study.

Our proposal is to use the marginal probabilities to define the joint distribution of two random variables. Different joint distributions will yield different mutual information values according to our revised definition of mutual information. We first established a few characteristics of the newly defined mutual information, and then we suggested a machine learning calculation approach. Lastly, we used credit rating with our newly specified mutual information.

References

1. Tourassi GD, Frederick ED, Markey MK, Floyd CE Jr. Application of the mutual information criterion for feature selection in computer-aided diagnosis. Med Phys. 2001;28(12):2394-2402.
2. Guyon I, Elisseeff A. An introduction to variable and feature selection. J Mach Learn Res. 2003;3:1157-1182.
3. Shannon CE. A mathematical theory of communication. Bell Syst Tech J. 1948;27(3):379-423.
4. Pinsker MS. Information and Information Stability of Random Variables and Processes. Academy of Science, USSR; c1960. (English translation by A. Feinstein, Holden-Day, San Francisco, USA, 1964).
5. Braga I. A constructive density-ratio approach to mutual information estimation: experiments in feature selection. J Inf Data Manag. 2014;5(1):134-143.
6. Verma RK. Information radius via Verma information measure in intuitionistic fuzzy environment. Int J Math Res. 2023;15(1):1-8.
7. Verma RK. Modified version of Verma measures of information and their kinship with past information measures. Int J Pure Appl Math Sci. 2023;16(1):17-24.
8. Verma RK. Obvious disparities on optimal guesswork wiretapper moments under mismatch related to non-Shannon cypher system. Asian J Prob Stat. 2023;24(4):17-22. DOI:10.9734/AJPAS/2023/v24i4531.
9. Verma RK. On optimal channel capacity theorems via Verma information measure with two-sided input in noisy state. Asian J Prob Stat. 2023;22(2):1-7. DOI:10.9734/AJPAS/2023/v22i2478.