Overheating allows you to push your modules beyond what there are suppose to do at risk of damaging the module's themselves and if left unchecked will burn the module out and make it unusable until you have docked and repaired.
Different modules give different benefits when overheated and for a better break down you can check the information on each module to get a better idea of what the bonus is but in short.
Weapons give increased damage, Propulsion gives extra speed and Propulsion Jamming gives extra range.
For the basics you need 'Thermodynamics' Skill which requires 'Power Grid Management V', 'Capacitor Management III' and 'Science IV' and most people I have spoken to agree that level 4 in Thermodynamics is a good place to start for PvP.
You can overheat the whole rack at once e.g. the low/mid/high rack or a module individually by pressing the green bar at the top of each module or by pressing the shift key and then selecting the module itself.
Inactive but overloaded modules do not generate heat until there are used and will benefit from the bonus on the first cycle, while an offline or empty module slot will function as a heat sink lowering the damage taken overtime both for the nearby modules and for the ship as a whole.
Jester's Trek Guide to overheating breaks it down more in-depth and is a good read for anyone that is new to overheating and I would suggest checking it out as the stuff I have put is a quick tour of overheating in comparison.
it seems outdated but gives you a good starting point and a good understanding of the principals of overheating while the Eve Online Wiki will bring you more up todate as Jester's Blog seems to talk about Heat Emission ratings while the modules no longer list that stat.
Overheating has its down sides and if not done correctly could leave you in a jam but done right can give you a needed boost when you need it.
or in my case it level the playing field a little while my skills catch up.