I love these questions, they are excellent thinking questions that barely give you enough information to solve them. In both cases it is probably very helpful to start by drawing the figures.
1) First draw a square then draw a circle that barely fits inside the square so it touches the sides of the square in four places.
2) Notice that the side length of the square is equal to twice the radius of the circle, therefore, the area of the square is equal to 20 in. times 20 in. or 400 inches squared.
3) The area of any circle is equal to its radius squared times pi. In this case, 10 squared times 3.14 which equals 314 square inches.
4) The difference between those two areas is 86 square inches.
Draw a regular hexagon and equilateral triangle to refer to then pick an easy side length for the equilateral triangle. I picked 10.
10 gives the triangle a perimeter of 30 so the hexagon also has a perimeter of 30 which means its sides equal 5.
Split the equilateral triangle down the middle to create two congruent right triangles. Use the pythagorean theorem to get the height of the triangles. If the base is 5 and the hypotenuse is 10 then the height is the square root of 10 squared minus 5 squared or 8.66. So the area of the triangle is 10 x 8.66 divided by two which equals 86.6 square units.
The area of the hexagon could be solved multiple ways but I like drawing three diagonals through it to make 6 equilateral triangles since we are already experts in the area of equilateral triangles. The side lengths of the triangles is 5.
Split one of those triangles down the middle and use the Pythagorean theorem:
The square root of 5 squared minus 2.5 squared is 4.33 so that is the height of the six triangles.
5 times 4.33 divided by 2 equals 10.825 and that is the area of one of the triangles. Then multiply that by 6 since there are six of them together in the hexagon: 64.95
Finally, the ratio would be 87:65.