Important: Use custom search function to get better results from our thousands of pages

Use " " for compulsory search eg:"electronics seminar" , use -" " for filter something eg: "electronics seminar" -"/tag/" (used for exclude results from tag pages)

Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
The Information Bottleneck: Theory and Applications
Post: #1

The Information Bottleneck: Theory and Applications

.pdf   Information Bottleneck:.pdf (Size: 1.34 MB / Downloads: 40)


This thesis introduces the first comprehensive review of the Information Bottleneck (IB) method along
with its recent extension, the multivariate IB. The IB method was originally suggested in [82] as a new
information-theoretic approach for data analysis. The basic idea is surprisingly simple: Given a joint distri-
μ, find a compressed representation of , denoted by , that is as informative as possible about
bution ́
. This idea can be formulated as a variational principle of minimizing the mutual information, ́
(which controls the compactness of the representation ), under some constraint on the minimal level of
μ . Hence, the fundamental trade-off between
mutual information that preserves about , given by ́
the complexity of the model and its precision is expressed here in an entirely symmetric form, where the
exact same concept of information controls both its sides. Indeed, an equivalent posing of the IB principle
would be to maximize the information maintains about , where the (compression) information ́
is constrained to some maximal level.
As further shown in [82], this constrained optimization problem can be considered analogous to rate distor-
tion theory, but with an important distinction: the distortion measure does not need to be defined in advance,
μ. Moreover, it leads to a tractable mathematical
but rather naturally emerges from the joint statistics, ́
analysis which provides a formal characterization of the optimal solution to this problem. As an immedi-
ate implication, the IB method formulates a well defined information-theoretic framework for unsupervised
clustering problems, which is the main focus of this thesis. Nonetheless, it is important to keep in mind
that the same underlying principle of a trade-off between information terms may have further implications
in other related fields, as recently suggested in [37].


In this chapter we provide a basic introduction to the remaining chapters. In the first section we present high
level descriptions of the fundamental trade-off between precision and complexity. One important variant of
this trade-off is formulated as the problem of unsupervised clustering, which is the main problem we address
in this thesis. In the next section we present the necessary preliminaries for our analysis. We conclude this
chapter by presenting a simple example in order to elucidate the central ideas that will be discussed later on.


In this section we introduce the basic concepts required for the next chapters. We start with some nota-
divergence and
tions, and further state the definitions of entropy, mutual and multi information,
divergence. Most of this section is based on Chapter 3⁄4 in [20], Chapter ¿ in [5] (which provides a friendly
introduction to the concept of entropy), and a work in progress by Nemenman and Tishby [55], which
introduces a new axiomatic derivative of mutual and multi-information.

Entropy and related concepts

Consider the following situation. We are given a finite collection of documents, denoted by
A person chooses to read a single document out of this collection, and our task is to guess which document
was chosen. Without any prior knowledge, all guesses are equally likely. We now further assume that we
3⁄4 ,
have access to a definite set of (exhaustive and mutually exclusive) probabilities, denoted by ́ μ
for all the possible choices. For example, let us assume that longer documents are more probable than shorter
ones. More specifically, that the probability of choosing each document is proportional to the (known) num-
ber of words that occur in it. If all the documents consist of exactly the same number of words, ́ μ is
uniform and obviously we are back at the starting point where no guess is preferable. However, if one docu-
ment is much longer than all the others, ́ μ will have a clear peak for this document, hence our chances of
providing the correct answer will improve. How can we quantify the difference between these two scenarios
in a well defined way?

Mutual information and multi-information

Let us reconsider our previous example of trying to guess what document was chosen. However, we now
assume that we have access not only to the prior distribution ́ μ, but rather to a joint distribution of
with some other random variable, . For concreteness, if values correspond to all the possible document
values correspond to all the distinct words occurring in this document
identities, let us assume that
collection. Thus, more formally stated, we assume that we have access to the joint distribution ́
which indicates the probability that a random word position in the corpus is equal to 3⁄4
while the
document identity is 3⁄4 . 3
Post: #2
The bottleneck literally refers to the upper narrow part of a bottle. In engineering, it refers to a phenomenon in which the performance or capacity of an entire system is limited by a single or small number of components or resources.


Marked Categories : information bottleneck ppt,

Quick Reply
Type your reply to this message here.

Image Verification
Image Verification
(case insensitive)
Please enter the text within the image on the left in to the text box below. This process is used to prevent automated posts.

Possibly Related Threads...
Thread: Author Replies: Views: Last Post
  well eqqupid commercial offices information 11th projects Guest 0 0 08-03-2018 10:22 PM
Last Post: Guest
  information about automatic machines for 11th o c project Guest 2 1,120 04-03-2018 05:54 PM
Last Post: Guest
  khankam marathi information Guest 0 0 23-02-2018 08:35 AM
Last Post: Guest
  killari earthquake information Guest 0 0 23-02-2018 12:10 AM
Last Post: Guest
  attending lectures on different geological techniques and geophysical techniques and subine 2 65 24-01-2018 12:34 AM
Last Post: RamiJer
  sound pollution information in marathi language pdf file Guest 3 5,654 12-01-2018 12:09 PM
Last Post: dhanabhagya
  mechanic diesel theory pdf in hindi medium Guest 1 133 08-01-2018 02:38 PM
Last Post: dhanabhagya
  marathi language information 11th commerce o c project Guest 0 0 03-01-2018 06:45 PM
Last Post: Guest
  information speed breaker in marathi Guest 1 0 28-11-2017 10:13 AM
Last Post: jaseela123
Heart collection of seals of different joint stock companies and the relevant information Guest 1 0 18-11-2017 11:32 AM
Last Post: jaseela123
This Page May Contain What is The Information Bottleneck: Theory and Applications And Latest Information/News About The Information Bottleneck: Theory and Applications,If Not ...Use Search to get more info about The Information Bottleneck: Theory and Applications Or Ask Here