There are many ways to control which objects end up in a selection set.
First, you can pick them one at a time by clicking directly on them.
Second, you can window over multiple objects. Windowing right-to-left (indicated by green shading, with a dashed outline) selects objects that fall at least partially within the window, while windowing left-to-right (shown with blue shading, and a solid outline) only selects objects that are completely within the window.
Third, you can de-select objects by holding down SHIFT and clicking or windowing over the objects. If you're in a command, like Move or Copy, you can also switch to a "remove objects" mode by typing R, in which case you don't need to hold SHIFT to remove objects. Typing A (not S) returns you to the "select objects" prompt.
Settings controlling some of these options are located on the Selection tab of the Options dialog box.