Homework 1

Due Friday January 24 at 5:00pm

Exercise 1

Let \(X_i \in \mathcal{X}\) for all \(i \in \{1, 2, \ldots\}\) and suppose our belief model for \(\mathbf{X} = \{ X_1, \ldots X_n \}\) is exchangeable for all \(n\). Show, using de Finetti’s theorem, that for all \(i \neq j\),

\[ Cov(X_i, X_j) \geq 0 \text{ and } \]

\[ Corr(X_i, X_j) \geq 0. \]

Exercise 2

Let \(X, Y, Z\) be random variables with joint density (discrete or continuous)

\[ p(x, y, z) \propto f(x,z) g(y, z) h(z). \]

Show that

  1. \(p(x | y, z) \propto f(x, z)\), i.e. \(p(x | y, z)\) is a function of \(x\) and \(z\).
  2. \(p(y | x, z) \propto g(y, z)\), i.e. \(p(y | x, z)\) is a function of \(y\) and \(z\).
  3. \(X\) and \(Y\) are conditionally independent given \(Z\).

Exercise 3

The number of particles \(Y\) emitted from a rock sample depends on the unknown amount \(\theta\) of the sample that is radioactive. For each possible value of \(\theta\),

\[ Pr(Y = y | \theta) = \theta^y e^{-\theta} / y! \] for each \(y \in \{0, 1, 2, \ldots \}\). Suppose it is known that the rock is one of three possible types \(A\), \(B\), or \(C\), each with a particular value of \(\theta\), that is, \(\theta \in \{\theta_A, \theta_B, \theta_C \}\) where \(\theta_A = 1.1\), \(\theta_B = 3.2\), and \(\theta_C = 4.5\).

  1. Make a graph of \(Pr(Y = y | \theta)\) as a function of \(y\) for each of the three possible values of \(\theta\) (for some reasonable range of \(y\)-values.

  2. Now suppose that the rock is of type \(A\), \(B\), or \(C\) with probabilities \(.4\), \(.3\) and \(.3\) respectively. Compute the marginal probability of \(Y\), that is,
    \(Pr(Y = y) = Pr(Y = y | \theta_A) Pr(\text{type} = A) + Pr(Y = y | \theta_B) Pr(\text{type} = B) + Pr(Y = y | \theta_C) Pr(\text{type} = C)\)
    Plot this as a function of \(y\) and compare the graph to the three graphs from part a.

  3. Suppose it is observed that \(Y = 4\). Using the rules of conditional probability, compute the probabilities of each type conditional on \(Y = 4\), that is, compute \(Pr(\theta = \theta_X | Y = 4)\) for each \(X \in \{A, B, C \}\). Compare these probabilities to the prior probabilities (.4, .3, .3).

Exercise 4

Let \(Y_1, \ldots Y_n\) be binary random variables that are conditionally independent given a value of a parameter \(\theta\), so that \(Pr(Y_i = 1 | \theta) = \theta = 1 - Pr(Y_i = 0 | \theta)\).

  1. Let \(y_1, \ldots y_n\) be a binary sequence so that \(y_i \in \{0, 1 \}\), for each \(i = 1, \ldots, n\). Using the rules of probability, derive a formula for \(Pr(Y_1 = y_1, \ldots, Y_n = y_n | \theta)\) as a function of \(y_1, \ldots y_n\) and \(\theta\). Simplify as much as possible.

  2. Let \(X = \sum_{i=1}^n Y_i\), what is the probability \(p(X = x | \theta)\) for any \(x \in \{0, 1, \ldots n\}\)? Explain (in 1-2 sentences) why \(n \choose x\) is in the expression.

  3. Compute and compare, using the definition of expectation and variance: \(E(Y_i|\theta)\), \(Var(Y_i|\theta)\) to \(E(X|\theta)\) and \(Var(X|\theta)\).

  4. Using calculus or otherwise, for a given value of \(x\), find “\(\theta_{MLE}\)”, the value of \(\theta\) that maximizes \(Pr(X = x|\theta)\). This value is called the “maximum likelihood estimator” or MLE, of \(\theta\).