a) Let $M=-y^{2}sin(xy),N=-xysin(xy)+2cos(xy)+3y,$
$My=-2ysin(xy)-xy^{2}cos(xy), Nx=-ysin(xy)-xy^{2}cos(xy)-2ysin(xy).$ Should be $M_y$ and so on
Let $R_{1}=\frac{My-Nx}{M}=\frac{ysin(xy)}{-y^{2}sin(xy)}=-\frac{1}{y},$
$\mu=e^{-\int R_{1}dy}=e^{\int\frac{1}{y}dy}=e^{ln(y)}=y.$
multiply each side by y,
$-y^{3}sin(xy)+(-xy^{2}sin(xy)+2ycos(xy)+3y^{2})y'=0.$
$My=-3y^{2}sin(xy)-xy^{3}cos(xy),Nx=-3y^{2}sin(xy)-xy^{3}cos(xy)=My.$
Thus, there exist $\varphi(x,y) such that \varphi x=M,\varphi y=N.$
$\varphi=y^{2}cos(xy)+h(y),\varphi y=2ycos(xy)-xy^{2}sin(xy)+h'(y)=-xy^{2}sin(xy)+2ycos(xy)+3y^{2}.$
$h'(y)=3y^{2},h(y)=y^{3}+C.$
Thus, $y^{2}cos(xy)+y^{3}=C.$
b) $y(\frac{\pi}{3})=1,cos(\frac{\pi}{3})+1=C.$
$C=\frac{3}{2},y^{2}cos(xy)+y^{3}=\frac{3}{2}.$