Imagery is a very broad term that refers to
the use of language to represent objects, actions, feelings, thoughts, ideas, states of mind and any sensory or extra-sensory experience.
(Cuddon, page 442-443)
Different books define different distinctions between literary terms. For example, Cuddon makes a distinction between literal, perceptual and conceptual imagery, using lines from Robert Lowell's Our Lady of Walsingham as illustration:
There once the penitents took off their shoes
And then walked barefoot the remaining mile;
And the small trees, a stream and hedgerows file
Slowly along the munching English lane,
Like cows to the old shrine, until you lose
Track of your dragging pain.
The stream flows down under the druid tree,
Shiloah’s whirlpools gurgle and make glad
The castle of God. (...)
- The first two lines use a literal image, in this case a visual one.
- The third line presents a perceptual image: trees, a stream and hedgerows don't literally move in a file, they happen to be arranged in what looks like a file to the casual observer.
- In the last line, the "castle of God" is a conceptual image: one can't really visualise it "but one may have an idea of it".
Imagery is very often conveyed using figurative language, such as metaphor, simile, synecdoche, onomatopoeia or metonymy.
Images needn't be visual, such as the one from Lowell's poem, above, but may also be olfactory, tactile, auditory, gustatory or abstract.
Peter Redgrove's poem "Lararus and the Sea" combines many examples:
The tide of my death came whispering [auditory] like this
Soiling [tactile, olfactory or both] my body with its tireless voice.
I scented the antique moistures when they sharpened
The air of my room, made the rough wood of my bed [tactile], (most dear),
Standing out like roots in my tall grave. [visual]
The symbol is more specific, even though its definition appears to have evolved over time (see e.g. Arnold and Sinemus, pages 193, 369). Simply put, a symbol is (a reference to) an object whose meaning goes beyond its material aspect.
At the most basic level, symbols can be subdivided into conventional symbols and private symbols (Klarer, page 55). Examples of conventional symbols include the the Stars and Stripes (which is not simply a colourful piece of fabric), the cross (not just two pieces of wood) and scales (to represent justice). An example of a private symbol is the albatros in Samuel Taylor Coleridge's The Rime of the Ancient Mariner. The Stars & Stripes, the cross and the albatros are all visual and examples of imagery, but they point to something beyond themselves that the reader must somehow decode. This is relatively straightforward for conventional symbols generated by one's own culture, though by no means always. For example, in the visual realm, medieval viewers were presumably familiar with the symbol of the lily in paintings such as Jan Van Eyck's Annunciation but modern viewers tend to see the lily as irrelevant until told what it represents. 
Below are a few other examples of symbols.
Wilt thou be gone? it is not yet near day:
It was the nightingale, and not the lark,
That pierced the fearful hollow of thine ear;
(Romeo and Juliet, Act 3, scene 5)
(The nightingale is a night singer and here symbolizes the night, whereas the lark represents the morning.)
In the medieval morality play Everyman, the main character symbolises the sinful Christian.
In John Keats's Ode on a Grecian Urn, the urn's round shape has been said to symbolise the closedness of the ideal poem (at least for the New Critics, see Klarer, page 21).
In the sentence "I am trying to plunge into the whirlpool of challenges", the phrase "whirlpool of challenges" is a perceptual image. However, if the text said that you fell into a whirlpool and the reader had to figure out that the whirlpool represents the many challenges you are facing, we would call it a (private) symbol.
- Cuddon, J. A.: The Penguin Dictionary of Literary Terms and Literary Theory. Third edition. Penguin, 1992.
- Klarer, Mario: Einführung in die anglistisch-amerikanistische Literaturwissenschaft. Darmstadt: Wissenschftliche Buchgesellschaft, 1994.
- Arnold, Heinz Ludwig; Sinemus, Volker (editors): Grundzüge der Literatur- und Sprachwissenschaft. Band 1: Literaturwissenschaft. Sixth edition. München: Deutscher Taschenbuch Verlag, 1980.
- Fricke, Harald; Zymner, Rüdiger: Einübung in die Literaturwissenschaft: Parodieren geht über studieren*. UTB 1616. Paderborn: Schöningh, 1991.
 I am leaving it unexplained on purpose.