I have not found that many geocaches so far (but my list is constantly growing), but I already have a few really necessary tools that I carry with me. Despite some good computer tools to plan your route and put the coordinates onto your GPS device other things are quite useful.
Of course you need a GPS device. As a start you can also use a smartphone (a geocaching app is available for different types).
Make yourself a profile on a geocaching site. I recommend geocaching.com to you: They offer lots of caches all over the world, the site design and layout is good and even with the free basic account you have nearly all you need. Just take some good additional computer tools to are able to plan your tour (e.g. bookmarks are just for premium members). I can recommend you thisadditional software tools.
You should take your own pen with you. Some caches provide one, but not all. You can also make a stemp with your nickname and a picture (I finally brought one myself and I am really happy with my choice). Perhaps think about buying a pen that is able to work in difficult situations (very cold, wet paper). There are also pens with integrated carabiner.
Then you should bring items to trade. Coins, buttons or keychain, you can trade anything you like. But be fair and trade equal!
Especially for town caching a forceps is good to have! Luckily I put mine into my caching bag just before I needed one to get the geocache.
If you are not very tall, you should have a look ont the terrain information or go caching with a taller person. When I think about even a telescopic pointer might work. *MustBuyOneSoon*
For solving the riddled you should start with the easy ones to get a feeling for the riddles. You can also ask a more experienced cacher to show you some basic tricks. If you don’t know one look out for cacher meetings near you. They are hold frequently (here on a monthly base) and even new cacher are highly welcome. Of course you can also use the forum on the geocaching.com site to get some help and chat with other cachers.
For night caching cat’s eyes or illuminating bracelets might be good to have.1