I'm assuming that you have to concatenate the toothpicks one after the other to build the sides of the rectangle.
Recalling that the opposite sides of a rectangle have the same length, the formula for its perimeter is
[tex] p = 2w+2h = 2(w+h) [/tex]
Since we have 12 toothpicks, we have
[tex] 2(w+h) = 12 \implies w+h = 6 [/tex]
So, the problem reduces to finding all the possible couples [tex] (w,h) [/tex] such that [tex] w > 0, h > 0, w+h = 6 [/tex]
It's easy to list all the possibilities:
[tex] w = 1,\quad h = 5 \implies \text{a 1-by-5 rectagle} [/tex]
[tex] w = 2,\quad h = 4 \implies \text{a 2-by-4 rectagle}[/tex]
[tex] w = 3,\quad h = 3 \implies \text{a 3-by-3 rectagle (thus a square)}[/tex]
The reimaining possibilities are just the reflected versions of those we already listed, so the 1-by-5 rectagle yields the existance of the 5-by-1 rectangle, and so on. It may even be debatable is these two cases should be considered as the same or not.
So, there are five possible rectangles in total (one of which is a square):
1-by-5
2-by-4
3-by-3
4-by-2
5-by-1