While antimalware tools will do most of the heavy lifting on bot detection, technically sophisticated users and certainly system administrators can analyze their machines to try to look for anomalies. I've written several articles on how to do this, including this detailed one on how to find malware on your Windows box.
One of the most useful tools in our arsenal is the humble netstat command. When run at a Windows command line with the "--nao 1" option, it will show all TCP and UDP port activity on a machine, displaying the process ID number every second. Because bots need to communicate with their botnet controller or peer-to-peer network; this technique can be used to look for unexpected communicating sessions on machines to identify a bot.
This was first published in February 2008