Interesting problem!
P(x)=x^3+3*x^2-2Ax+3
is a third degree polynomial.
The divisor,
D(x)=x^2+1
is a second degree polynomial.
We are given
P(x)/D(x) has a remainder of R(x)= -5x, note that the constant term of the remainder is zero.
The fact that the the quotient Q(x) is a linear expression, and constant term of R(x) equals zero allows us to deduce that the quotient equals Q(x)=x^3/x^2 + 3/1 = x+1, i.e. the sum of the quotients of the leading coefficient divided by that of D(x), or x^2, and the constant term divided by the constant term of D(x), or 1.
Using Q(x)=x+1, the product of the divisor D(x) and the quotient Q(x) added to the remainder R(x) should be equal to the original polynomial P(x), i.e.
P(x) = D(x)*Q(x)+R(x)
=(x^2+1)(x+3)-5x
=x^3+3x^2-4x+3
(To check, divide (using synthetic division) P(x) by D(x) and we should get R(x))
Now solve for A
-2A = -4
A = (-4)/(-2) = 2
