I have recently taken an interest in web crawling and am aware of robots.txt guidelines, but I was interested in the specifics of what is considered "polite" in terms of pages processed per second and the number of threads that should be set crawling a single website at any given time. Any general guidelines are greatly appreciated as I don't want to be impolite, destructive, or risk getting blocked by a website.
Also, I realize different websites will experience different amounts of strain based on my crawling, so any advice on how I can properly gauge this and account for it is welcome.