You did correctly (except renaming $u$ to $\mu$; so I replaced (batch) $\mu$ by $u$) until the last equation.
The primary purpose is not to find $u$ but to determine where it is a solution.
Follow Hint provided: your (\ref{K}) is my (\ref{eq-3}). Write system (\ref{eq-3})-- (\ref{eq-4}) and solve it which gives the boundary of the required domain.
More clear what is the core of the problem will be after Lecture Wed, Sep 23 (see also Subsection 2.1.6 Quasilinear equations (http://www.math.toronto.edu/courses/apm346h1/20159/PDE-textbook/Chapter2/S2.1.html#sect-2.1.6))
---
So according to (\ref{K}) $F(x,t,u)=t^2 u^2+2(tx-1) u+x^2$ and system (\ref{eq-3})-- (\ref{eq-4}) is
\begin{align}
&t^2 u^2+2(tx-1) u+x^2,\label{L}\\
&2t^2u +2 (tx-1) =0.\label{M}
\end{align}
(\ref{M}) implies $u= -(tx-1)t^{-2}$ and plugging to (\ref{L}) we get $x=1/(2t)$. This is a border of the domain where $u$ is defined. See attached picture with cyan characteristics. Their continuations after they touched $x=1/(2t)$ should be ignored as at the point of tangency equation breaks (as it does not belong to domain).
(http://www.math.toronto.edu/courses/apm346h1/20159/Forum/W1.svg)