Capture stats about hosts on status stream post fetching

[jnioche]

This could be used to notify external components of host-related information such as:

• robots delay settings
• fetch times
• HTTP 429 codes

This would be used by the FetcherBolt, for instance, to implement throttling at the spout level and users would not have to modify the Fetcher code for doing so.

@owenrh

[jnioche]

The fetch times are stored in the metadata - the same could be done with the robots delay. Instead of having yet another stream, we could have a custom bolt intercepting the URLs from the FetcherBolt on the status stream and doing whatever it wants based on the content of the metadata.

[jnioche]

Having a windowing bolt for that purpose would make it easier to aggregate counts per minute

[jnioche]

Could also rely on the state framework provided by Storm to send to a pluggable storage e.g. Redis

https://storm.apache.org/releases/1.2.2/Windowing.html
https://storm.apache.org/releases/1.2.2/State-checkpointing.html