Latest Policy on API posts/all request frequency?

[Jerry Krinock]

I have designed my application to prevent a user from requesting posts/
all to often and being banned or throttled. Of course, this does not
happen very often to a real user, because my application caches the
data and checks with posts/update, doing posts/all only when needed.
But it can happen if the user does certain things, and of course I can
inadvertently send lots of posts/all during development :(

A couple years ago, I noted that if I sent posts/all more frequently
than once a minute, I would get a 503 response code and, apparently,
my IP address would be banned for a couple hours. This has now changed.

Even sending posts/all less frequently than once every 3 minutes, I
can still get banned, although now it appears to be for only five
minutes or so. Another thing different is that instead of getting a
503 response code, I get a 999 response code and a web page saying
"We're sorry, we couldn't complete your request right now."

So, if someone could please tell me what the latest policy is I'd be
happy to design for it, with plenty of margin.

Also, is it true that the "You've been sending too many posts/all"
response is now 999 instead of 503? The documentation still looks the
same as it did two years ago, implying that throttling is indicated by
a 503...

http://delicious.com/help/api

Thanks,

Jerry Krinock

P.S. For future reference, I believe that a 4xx Client Error would be
more appropriate than 999.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

According to the last paragraph in sec 6.1.1,

http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html#sec6.1

the Status Codes are supposed to be "extensible". I guess that means
you could make up your own, maybe 450 or something. 999 looks like
not a lot of thought went into it ;)

[Chris Draycott]

Hi Jerry,

I'll do my best to answer your problem :)
First of all, the 999 is the Yahoo-wide response code for this type of rate limiting in the message, but in the case of Delicious you should actually be getting a 500 http status code (I checked our logs, and that's what we're logging, at least). Still, as you point out, the docs say 503, and it's not that. While 500 does not quite meet the exact definition of what we're doing, we use 503 for other cases and wanted to have a distinct code to allow them to be differentiated.

As for the rate at which you can hit posts/all, which I can't give you the exact details because they are always open to tweaking depending on abuse vectors etc, I would say that posts/all as a raw query should be avoided wherever possible. If, however, you request from that api using "results" parameter (probably in conjunction with "start") you can make many more requests. Depending on what you're trying to do, I would suggest that results=100 is probably a reasonable query, and the "paginate" until you get all the results.For most users, this will only take a few queries to get the whole collection.

Hope this helps
Chris

Post by Jerry Krinock
I have designed my application to prevent a user from requesting posts/
all to often and being banned or throttled. Of course, this does not
happen very often to a real user, because my application caches the
data and checks with posts/update, doing posts/all only when needed.
But it can happen if the user does certain things, and of course I can
inadvertently send lots of posts/all during development :(
A couple years ago, I noted that if I sent posts/all more frequently
than once a minute, I would get a 503 response code and, apparently,
my IP address would be banned for a couple hours. This has now changed.
Even sending posts/all less frequently than once every 3 minutes, I
can still get banned, although now it appears to be for only five
minutes or so. Another thing different is that instead of getting a
503 response code, I get a 999 response code and a web page saying
"We're sorry, we couldn't complete your request right now."
So, if someone could please tell me what the latest policy is I'd be
happy to design for it, with plenty of margin.
Also, is it true that the "You've been sending too many posts/all"
response is now 999 instead of 503? The documentation still looks the
same as it did two years ago, implying that throttling is indicated by
a 503...
http://delicious.com/help/api
Thanks,
Jerry Krinock
P.S. For future reference, I believe that a 4xx Client Error would be
more appropriate than 999.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
According to the last paragraph in sec 6.1.1,
http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html#sec6.1
the Status Codes are supposed to be "extensible". I guess that means
you could make up your own, maybe 450 or something. 999 looks like
not a lot of thought went into it ;)

[Jerry Krinock]

Post by Chris Draycott
but in the case of Delicious you should actually be getting a 500
http status code (I checked our logs, and that's what we're logging,
at least)...
Hope this helps

Yes, that helps, Chris. Indeed, posts/all is rarely sent, but I'll
study using the "start" and "results" parameters. It would be nice to
download in chunks so that I can display progress information.

Just for your information, though, here is a console transcript
showing the 999 responses (not 500). You see I requested posts/all
six times, by hitting the up-arrow and return keys. The first four
requests produced 200 responses, then after that, I started getting
999. Actual data output was sent to /dev/null, and I replaced out my
actual password before pasting.

Thank you,

Jerry


Jerrys-Mac-Mini: jk$ curl -ujerrykrinock4:myPassword -w "Received HTTP
Status Code: %{http_code}\n" -o "/dev/null" https://api.del.icio.us/v1/posts/all
% Total % Received % Xferd Average Speed Time Time
Time Current
Dload Upload Total Spent
Left Speed
100 914 0 914 0 0 1252 0 --:--:-- --:--:--
--:--:-- 0
Received HTTP Status Code: 200
Jerrys-Mac-Mini: jk$ curl -ujerrykrinock4:myPassword -w "Received HTTP
Status Code: %{http_code}\n" -o "/dev/null" https://api.del.icio.us/v1/posts/all
% Total % Received % Xferd Average Speed Time Time
Time Current
Dload Upload Total Spent
Left Speed
100 914 0 914 0 0 1362 0 --:--:-- --:--:--
--:--:-- 0
Received HTTP Status Code: 200
Jerrys-Mac-Mini: jk$ curl -ujerrykrinock4:myPassword -w "Received HTTP
Status Code: %{http_code}\n" -o "/dev/null" https://api.del.icio.us/v1/posts/all
% Total % Received % Xferd Average Speed Time Time
Time Current
Dload Upload Total Spent
Left Speed
100 914 0 914 0 0 1416 0 --:--:-- --:--:--
--:--:-- 0
Received HTTP Status Code: 200
Jerrys-Mac-Mini: jk$ curl -ujerrykrinock4:myPassword -w "Received HTTP
Status Code: %{http_code}\n" -o "/dev/null" https://api.del.icio.us/v1/posts/all
% Total % Received % Xferd Average Speed Time Time
Time Current
Dload Upload Total Spent
Left Speed
100 914 0 914 0 0 1408 0 --:--:-- --:--:--
--:--:-- 0
Received HTTP Status Code: 200
Jerrys-Mac-Mini: jk$ curl -ujerrykrinock4:myPassword -w "Received HTTP
Status Code: %{http_code}\n" -o "/dev/null" https://api.del.icio.us/v1/posts/all
% Total % Received % Xferd Average Speed Time Time
Time Current
Dload Upload Total Spent
Left Speed
100 4233 0 4233 0 0 6838 0 --:--:-- --:--:--
--:--:-- 217k
Received HTTP Status Code: 999
Jerrys-Mac-Mini: jk$ curl -ujerrykrinock4:myPassword -w "Received HTTP
Status Code: %{http_code}\n" -o "/dev/null" https://api.del.icio.us/v1/posts/all
% Total % Received % Xferd Average Speed Time Time
Time Current
Dload Upload Total Spent
Left Speed
100 4233 0 4233 0 0 6712 0 --:--:-- --:--:--
--:--:-- 217k
Received HTTP Status Code: 999
Jerrys-Mac-Mini: jk$