Use the property that the scalar product of two vectors that are orthogonal must be zero. We call the vector that we are looking for \(\left|{\phi_{1}^{'}}\right\rangle = \alpha\left|{+}\right\rangle + \beta\left|{-}\right\rangle \). So we want to take the braket of \(\left|{\psi_{1}^{'}}\right\rangle \) and \(\left|{\phi_{1}^{'}}\right\rangle \) Since we are trying to find \(\alpha\) and \(\beta\), it is easier to make \(\left|{\phi_{1}^{'}}\right\rangle \) be the bra so that we do not need to complex conjugate the coefficients that we are trying to determine. \begin{align*} \left\langle {\psi_1 }\middle|{ {\phi}_{1}^{'} }\right\rangle =& \Big( \frac{1}{\sqrt{3}} \left\langle {+}\right| + (-i) \frac{\sqrt{2}}{\sqrt{3}} \left\langle {-}\right| \Big) \Big( \alpha\left|{+}\right\rangle + \beta\left|{-}\right\rangle \Big)\\ =& \frac{\alpha}{\sqrt{3}} - \beta \frac{i\sqrt{2}}{\sqrt{3}}\\ = 0 \end{align*}
Then we can choose \(\alpha = \beta \,i\sqrt{2}\), and \(\alpha = i\sqrt{2}, \ \beta=1\) (for example). There are an infinite number of other choices that we can make that differ from this one by an overall scale. This is because the vector we are finding is orthogonal, but not necessarily normalized. Also, multiplication by an overall phase will not affect the orthogonality. Find the norm of our choice of vector, \begin{align*} \left\langle {\phi_{1}^{'}}\middle|{\phi_{1}^{'}}\right\rangle =& \left(\!\!\begin{array}{cc} -i\sqrt{2} & 1 \end{array}\!\!\right) \left(\!\!\begin{array}{c} i\sqrt{2} \\ 1 \end{array}\!\!\right)\\ =& (-i\sqrt{2}) \, i\sqrt{2} + 1 = 3, \sqrt{\left\langle {\phi_{1}^{'}}\middle|{\phi_{1}^{'}}\right\rangle }\\ =& \sqrt{3}. \end{align*}
The column--vector representation for \(\phi_1^{'}\) is in the \(\big\{\left|{+}\right\rangle ,\left|{-}\right\rangle \big\}\) basis. Normalized, \begin{align*} \left|{\phi_1}\right\rangle =& \frac{1}{\sqrt{3}} \Big( i\sqrt{2} \left|{+}\right\rangle + \left|{-}\right\rangle \Big),\mathrm{or} \\ \left|{\phi_1}\right\rangle \stackrel{\cdot}{=}& \frac{1}{\sqrt{3}} \left(\!\!\begin{array}{c} i \sqrt{2} \\ 1 \end{array}\!\!\right), \ \mathrm{in the} \ \Big\{ \left|{+}\right\rangle ,\left|{-}\right\rangle \Big\} \ \mathrm{basis}. \end{align*}
In this case, let us do the problem in the matrix notation also, for comparison. \begin{align*} \left\langle {\phi_{1}^{'}}\middle|{\psi_1}\right\rangle =& \left(\!\!\begin{array}{cc} 1/\sqrt{3} & -i\sqrt{2}/\sqrt{3} \end{array}\!\!\right) \left(\!\!\begin{array}{c} \alpha \\ \beta \end{array}\!\!\right)\\ =& \frac{\alpha}{\sqrt{3}} - \beta \frac{i\sqrt{2}}{\sqrt{3}} = 0 \\ \Rightarrow& \ \alpha = \beta \, i \sqrt{2}. \end{align*}
For the other two states the calculation goes the same way. \begin{align*} \left\langle {\psi_2 }\middle|{ {\phi}_{2}^{'} }\right\rangle =& \Big( \frac{1}{\sqrt{5}} \left\langle {+}\right| - \frac{2}{\sqrt{5}} \left\langle {-}\right| \Big) \Big( \alpha\left|{+}\right\rangle + \beta\left|{-}\right\rangle \Big) \\ =& \frac{\alpha}{\sqrt{5}} - \frac{2 \beta }{\sqrt{3}} \\ =& 0 \end{align*}
So \(\alpha = 2\beta\), and \(\alpha = 2, \beta=1\) satisfy the condition of orthogonality, thus \(\left|{\phi_{2}^{'}}\right\rangle = 2\left|{+}\right\rangle + \left|{-}\right\rangle \). Normalized, \begin{align*} \left\langle {\phi_{2}^{'}}\middle|{\phi_{2}^{'}}\right\rangle =& 5, \mathrm{and} \\ \left|{\phi_2}\right\rangle =& \frac{1}{\sqrt{5}} \Big( 2 \left|{+}\right\rangle + \left|{-}\right\rangle \Big) \ = \ \frac{1}{\sqrt{5}} \left(\!\!\begin{array}{c} 2 \\ 1 \end{array}\!\!\right) \end{align*}
Remember that the matrix notation denotes a representation in some basis.
It should be clear from the context which basis one is working with, and when it isn't the representation should be stated explicitly. (In the last term above, the column--vector for \(\left|{\phi_2}\right\rangle \), is clearly the matrix representation in the basis \(\big\{\left|{+}\right\rangle ,\left|{-}\right\rangle \big\}\).) Often one uses the symbol \(\stackrel{\cdot}{=}\) to remind ourselves that we are representing a vector in a particular basis. For the case \(\psi_3\), we first note some handy manipulations. \begin{equation} \label{eq-ComplexIden} i \, e^{ i\frac{\pi}{4} } = i \, \big( \cos{\frac{\pi}{4}} + i \sin{\frac{\pi}{4}} \big) = i\, \frac{\sqrt{2}}{2} \big( 1 + i \big) = \frac{\sqrt{2}}{2} \big( -1 + i \big) = \frac{ -1 + i }{\sqrt{2}} . \end{equation} (Once we are at it, also note that \(i = e^{i \pi/2}\), so: \( \displaystyle i \, e^{ i\frac{\pi}{4} } = e^{ i\frac{\pi}{2} } \, e^{ i\frac{\pi}{4} } = e^{ i \, (\frac{\pi}{2} + \frac{\pi}{4}) } = e^{ i\frac{3\pi}{4} } \).) \begin{align*} \left\langle {\psi_3 }\middle|{ {\phi}_{3}^{'} }\right\rangle =& \Big( \frac{1}{\sqrt{2}} \left\langle {+}\right| + (- i) \frac{e^{(-i) \frac{\pi}{4}}}{\sqrt{2}} \left\langle {-}\right| \Big) \Big( \alpha\left|{+}\right\rangle + \beta\left|{-}\right\rangle \Big)\\ =& \frac{\alpha}{\sqrt{2}} - \beta \, \frac{i \,e^{-i \frac{\pi}{4} } }{\sqrt{2}} \\ = 0 \end{align*}Now \(\beta = 1\), \(\alpha = i \,e^{-i \frac{\pi}{4}} = \frac{1}{\sqrt{2}}\, (1 + i)\) (the above identity with \(-i\) in the exponent). Then, \begin{align*} \left\langle {\phi_{3}^{'}}\middle|{\phi_{3}^{'}}\right\rangle =& \left(\!\!\begin{array}{cc} \frac{1 -i }{\sqrt{2}} & 1 \end{array}\!\!\right) \left(\!\!\begin{array}{c} \frac{1 + i }{\sqrt{2}} \\ 1 \end{array}\!\!\right) \\ =& \Big( \frac{1 -i }{\sqrt{2}} \Big) \, \Big( \frac{1 + i }{\sqrt{2}} \Big) + 1. \end{align*}
The difference of squares is convenient to use here, \begin{align*} \frac{(1 -i ) \, (1 +i ) }{2} + 1 =& \frac{ (1)^2 - (i^2) }{2} + 1 = 2, \mathrm{and} \\ \left|{\phi_3}\right\rangle =& \frac{1}{\sqrt{2}} \left(\!\!\begin{array}{c} \frac{1+i }{ \sqrt{2} } \\ 1 \end{array}\!\!\right). \end{align*}
Recall that \(\left\langle {\psi_i}\middle|{\psi_j}\right\rangle = \left\langle {\psi_j}\middle|{\psi_i}\right\rangle ^{*}\), and here we only calculate one set of products. \begin{align*} \left\langle {\psi_1}\middle|{\psi_2}\right\rangle =& \Big( \frac{1}{\sqrt{3}}\left\langle {+}\right| - i\frac{\sqrt{2}}{\sqrt{3}}\left\langle {-}\right| \Big) \Big( \frac{1}{\sqrt{5}}\left|{+}\right\rangle - \frac{2}{\sqrt{5}}\left|{-}\right\rangle \Big)\\ =& \frac{1}{\sqrt{15}} + i\frac{2\sqrt{2}}{\sqrt{15}}\\ =& \frac{1+i\,2\sqrt{2}}{\sqrt{15}}. \end{align*}
For calculations involving \(\psi_3\) we'll use the identities of Eq. (\ref{eq-ComplexIden}) from part (a). \begin{align*} \left\langle {\psi_1}\middle|{\psi_3}\right\rangle =& \Big( \frac{1}{\sqrt{3}}\left\langle {+}\right| - i\frac{\sqrt{2}}{\sqrt{3}}\left\langle {-}\right| \Big) \Big( \frac{1}{\sqrt{2}}\left|{+}\right\rangle + \frac{\left(\frac{-1 + i}{\sqrt{2}}\right) }{\sqrt{2}}\left|{-}\right\rangle \Big) \\ =& \frac{1}{\sqrt{6}} + \frac{i+1}{\sqrt{6}}\\ =& \frac{2+i}{\sqrt{6}} \end{align*}
The last product for \(i\neq j\) that we need to calculate, \begin{align*} \left\langle {\psi_2}\middle|{\psi_3}\right\rangle =& \Big( \frac{1}{\sqrt{5}}\left\langle {+}\right| - \frac{2}{\sqrt{5}}\left\langle {-}\right| \Big) \Big( \frac{1}{\sqrt{2}}\left|{+}\right\rangle + \frac{\left(\frac{-1 + i}{\sqrt{2}}\right) }{\sqrt{2}}\left|{-}\right\rangle \Big)\\ =& \frac{1\!+\!\sqrt{2} - i\,\sqrt{2} }{\sqrt{10}} \end{align*}
The cases \(i=j\) are normalization checks. (Quantum mechanical states should in general be normalized.) If \(\left|{\psi_i}\right\rangle = \alpha_i\left|{+}\right\rangle +\beta_i\left|{-}\right\rangle \), then \(\left\langle {\psi_i}\middle|{\psi_i}\right\rangle \left|\alpha_i\right|^2 + \left|\beta_i\right|^2 = 1\): \begin{align*} \left\langle {\psi_1}\middle|{\psi_1}\right\rangle =& \left(\frac{1}{\sqrt{3}}\right)^2 + \left( \frac{i\sqrt{2}}{\sqrt{3}} \right) \left( \frac{-i\sqrt{2}}{\sqrt{3}} \right) = 1 \\ \left\langle {\psi_1}\middle|{\psi_1}\right\rangle =& \left(\frac{1}{\sqrt{5}}\right)^2 + \left(-\frac{2}{\sqrt{5}}\right)^2 = 1 \end{align*}
and \begin{align*} \left\langle {\psi_3}\middle|{\psi_3}\right\rangle =& \left(\frac{1}{\sqrt{2}}\right)^2 + \left( \frac{ie^{i\frac{\pi}{4}}}{\sqrt{2}} \right) \left( \frac{-i e^{-i\frac{\pi}{4}} }{\sqrt{2}} \right)\\ =& \frac{1}{2} + \frac{ (i)(-i) e^{(i-i)\frac{\pi}{4}} }{2} \\ =& \frac{1+e^{0}}{2} \\ =& 1 \end{align*}