10th
Graceful Degradation
Long long ago, at DoubleClick, we added to the DART ad server a feature called “dot mode”.
Basically it worked like this: on an ad request, if we have more than a certain # of concurrent threads active, return a 1x1 clear gif (and do no computation or logging). That is, if we are backlogging, don’t serve an ad.
One nuance with the above is that any load balancing system in front of the ad servers needs to know that a “dot” is an error.
Adding this little feature turned out to be a great move. It then became very hard to kill a server with transient load. Further, we get statistics on how things are working. “This server served 20 million ads and 3 dots.” We can look at the ratio and infer things. The ops mentality became a bit about watching for dots instead of watching for complete failures.
