h = (19 - sqrt(97))/6, which is approximately 1.525190366
The volume of the box will be
V = lwh
And l will be
a - 2h
And w will be
b - 2h
So using the above, the volume of the box will be
V = lwh
V = (a - 2h)(b - 2h)h
V = (11 - 2h)(8 - 2h)h
V = (88 - 22h -16h + 4h^2)h
V = (88 - 38h + 4h^2)h
V = 88h - 38h^2 + 4h^3
Since you're looking for a maximum, that screams "First derivative" So let's calculate the first derivative of the function and solve for 0.
V = 88h^1 - 38h^2 + 4h^3
V' = 1*88h^(1-1) - 2*38h^(2-1) + 3*4h^(3-1)
V' = 1*88h^0 - 2*38h^1 + 3*4h^2
V' = 88 - 76h + 12h^2
We now have a quadratic equation. So using the quadratic formula with A=12, B=-76, and C=88, calculate the roots as:
(19 +/- sqrt(97))/6
which is approximately 1.525190366 and 4.808142967
We can ignore the 4.808142967 value since although it does indicate a slope of 0, it produces a negative width and is actually a local minimum of the volume function.
So the optimal value of h is (19 - sqrt(97))/6, which is approximately 1.525190366
