To find critical points, let $x'=0, y'=0$
Thus, $-2x-y-x(x^2+y^2)=0, x-y+y(x^2+y^2)=0$
$ (0,0)$ is one crtical point
Also, $(0.330757,1.09242), (0.330757,-1.09242)$ are also critical points
Let $f(x)=-2x-y-x(x^2+y^2), g(x)=x-y+y(x^2+y^2)$
$f_x=-2-3x^2-y^2, f_y=-1-2xy$
Also, $ g_x=1+2xy, g_y=-1+x^2+3y^2$
$J=\begin{bmatrix}
-2-3x^2-y^2&-1-2xy\\
1+2xy&-1+x^2+3y^2\\
\end{bmatrix}$
Plug in the critical points to find eigenvalues of each linear system
For $(0,0)$,
We get $\begin{bmatrix}
-2&--1\\
1&-1\\
\end{bmatrix}$,
$r=\frac{-3\pm\sqrt{-3}i}{2}$
Therefore, it is stable at $(0,0)$
Similarly, for others critical points,
Plug in and eigenvalues are $-3.5092, 2.6771$
Thus, it is a saddle point therefore unstable