*Funny story: after I posted this I also checked my work against wolfram alpha, which said I got it wrong for the homogeneous part. I spent some time trying to spot any error, but there isn't any. I am smarter than the machine!*

(You can press "show step by step solution" and wolfram alpha will come up with what I came up with, so that's why I'm so confident. I filed a bug report, maybe someday this will be fixed...)

$\renewcommand{Re}{\operatorname{Re}}$

Here we use the method of undetermined coefficients. First we find the characteristic equation and its derivative (which will definitely be useful):

$$Q(r) = r^2 + 4r + 5$$

$$Q^\prime(r) = 2r + 4$$

Now we solve for $te^{-2t}$, for which our exponent is $r_1 = -2$.

$$Q(r_1) = 4 - 8 + 5 = 1$$

$$Q^\prime(r_1) = -4 + 4 = 0$$

So we want to use $t e^{-2t}$. If we plug that in we get $L[t e^{-2t}] = e^{-2t}(t Q(-2) + Q^\prime(-2)) = t e^{-2t}$. This is what we want, so we're done the first part.

Now we solve for $e^{-2t} cos(t)$, for which the exponent is $r_2 = -2 + i$.

$$Q(r_2) = (-2+i)^2 + 4(-2 + i) +5 = 4 - 4i - 1 - 8 + 4i + 5 = 0$$

$$Q^\prime(r_2) = 2(-2+i) + 4 = -4 + 2i + 4 = 2i$$

So we want to use $t e^{(-2 + i)t}$. If we plug that in, we get $L[t e^{(-2 + i)t}] = e^{(-2 + i)t}(t Q(-2 + i) + Q^\prime(-2 + i)) = (2i)e^{(-2 + i)t}$.

We don't want that 2i, so let's divide it out in the input. It turns out that $1/2i = i/-2 = -i/2$. So $L[\frac{-i}{2} t e^{(-2 + i)t}] = e^{(-2 + i)t}$, which is nearly right.

Note:

$$\Re e^{(-2 + i)t}) = \Re (e^{-2t}e^{it}) = \Re (e^{-2t}(\cos t + i \sin t)) = e^{-2t} \cos t$$.

Since $L$ is linear, if we take only the real part of the input, we'll get only the real part of the output. So we need to compute the real part of $\frac{-i}{2} t e^{(-2 + i)t}$.

To do that let's multiply it out:

$$\frac{-i}{2} t e^{(-2 + i)t}$$

$$\frac{-i}{2} t e^{-2t}(\cos t + i \sin t)$$

$$t e^{-2t}(\frac{-i}{2}\cos t + \frac{-i}{2}i \sin t)$$

$$t e^{-2t}(\frac{-i}{2}\cos t + \frac{1}{2}\sin t)$$

The real part of that is $\frac{1}{2}t e^{-2t}\sin t$, so that $L[\frac{1}{2}t e^{-2t}\sin t] = e^{-2t} \cos t$.

So then we put our two subproblem solutions together and we get a particular solution, thanks to the linearity of $L$. Our particular solution is:

$$Y(t) = t e^{-2t} + \frac{1}{2}t e^{-2t}\sin t$$

What remains is to compute a general solution for the homogenous part, and we can combine that to find the general solution for the non-homogenous ODE.

During the above work we found that $-2 + i$ was a root of the characteristic equation. So the conjugate is also a root, $-2 - i$, and we can take the real and imaginary parts of $e^{-2 \pm i}$ to find the solution, which is:

$$y_{gen[homogeneous]}(t) = c_1 e^{-2t}\cos(t) + c_2 e^{-2t}\sin (t)$$

So the general solution for the nonhomogeneous ODE is:

$$y(t) = c_1 e^{-2t}\cos(t) + c_2 e^{-2t}\sin (t) + t e^{-2t} + \frac{1}{2}t e^{-2t}\sin (t)$$