Information Theory

            Entropy is a measure of disorder, and disorder is essentially the same thing as ignorance. This is how entropy is related to information theory (Angrist & Hepler, 1967; Gell-Mann, 1994; Gleick, 1987).

            When we toss a coin, we have two possible outcomes. When we throw a die, we have six possible outcomes. According to Claude Shannon (1948), the founder of information theory, information refers simply to the number of possibilities, Z. So that information, I, is expressed as:

                        I = log₂ Z

            Shannon used the logarithm to the base 2 because modern communication works with binary numbers or bits. When the letters of a good novel are each counted up, the resultant totals for each letter are the letter frequencies for that particular book. Shannon let j = 1 for a, 2 for b, 3 for c, and so on, so that the relative frequency N_j would be the frequency of occurrence of these letters in a particular book. Then he calculated that the probability of finding any letter labeled j out of a total of N letters would be:

                         P_j = N_j/N

            From this he showed that the average information per letter contained in that book will be:

                        I = -3p_j log₂ p_j

                               ^j

            With one more step, in which he added a constant, K, he arrived at the formula for Shannon uncertainty, Shannon entropy, Shannon Information, or simply, information entropy (Çambel, p 147):

            This equation provides a measure of the disorder or ignorance that may exist in a quantity of information. However, Shannon information does not concern itself with meaning, and it only applies to closed systems. Shannon used his concept to study the capacity of a communication channel to transfer information even under the impact of noise (Haken, 1988).

            Information theory has been found useful in detecting and correcting errors as well as in data compression. The brain, for example, in order to be a detector or receiver of information, must be able to account for the length of the message as well as determine any hidden or decoded information in the message. In addition, in order for the brain to obtain meaning from any message, a shared interpretation of all information symbols used must first be established (Cohen & Stewart, 1994).

            Jung (1981) also equated the will with “disposable energy,” implying that energy can be stored within and dispensed from the psyche. According to information theory, in order for information to have meaning, there must be a sharing of symbolic coding. Jung’s collective unconscious allows psyches to share archetypal meanings. The archetypes, serving in their role as chaotic attractors, create upredictability and as demonstrated later, can either raise or lower the entropy in the ego. They attempt to balance the exchange of information, which must be assimilated by the ego in order to be meaningful.

            Jung (1981) wrote that “all knowledge is the result of imposing some kind of order upon the reactions of the psychic system as they flow into our consciousness” (p. 171).  Thus the imposition of order upon the chaotic flow of our sensory impressions is what gives rise to meaningful information.