Discourse API Rate Limit: I Got a 429 on Request One

The third request my census script sent to forum.openwrt.org this morning came back like this:

$ curl -s -A "$UA" -w '\n%{http_code}\n' 'https://forum.openwrt.org/search.json?q=test'
{"failed":"FAILED","message":"You’ve performed this action too many times, please try again later."}
429

That was the first search request in the run. Requests one and two — /about.json and /latest.json, one second apart — both returned 200. So did request four, a 5,182-post topic. Only search was throttled, and it was throttled before I had done anything.

Worse, the 429 body is valid JSON. A scraper that does json.loads(r.read()).get("topics", []) without checking the status code sees a well-formed response with zero results and reports that the forum has nothing on the subject. I have already shipped one tool that read a 429 as a real answer, so I went looking for the actual numbers instead of guessing.

The anonymous search limit is not yours

Discourse publishes its per-IP budget in Global rate limits and throttling: 200 requests per minute and 50 per 10 seconds, in block mode by default. Nothing I did came close to that.

Search has its own budget, and it is in config/site_settings.yml rather than the docs, because the settings are marked hidden:

rate_limit_search_anon_user_per_minute:     15
rate_limit_search_anon_user_per_second:      2
rate_limit_search_anon_global_per_minute:  150
rate_limit_search_anon_global_per_second:    8
rate_limit_search_user:                     30

Two of those five are global. They are not per IP and not per session — they are one shared bucket for every logged-out visitor of that forum. From outside I cannot see which bucket emptied, but the global per-second allowance is eight searches for the entire forum, and OpenWrt has a large anonymous readership. It does not take a crowd.

This is the part that changes how you write the client. A per-IP limit is a contract you can honour by slowing down. A shared global limit means your first request can fail through no fault of yours, and retrying politely does not help — I retried twice over the following hour, once with a different query, and got the same 429 both times. It is also why search is the one endpoint where an anonymous reader is worse off than a logged-in one: rate_limit_search_user is 30 a minute and it is per user.

Forum members hit the per-IP version of this without any script at all. The meta thread 429 error when opening multiple topics (11 posts, 3,233 views) is somebody opening tabs too fast, and How to avoid throttling limits with admin API key? (12 posts) is an operator still getting 429s after raising the admin ceiling to 600 a minute.

30 forums, 27 answered with no API key at all

Before writing a client I wanted to know how much of this is universal. So I built a census: 30 public Discourse hostnames, four requests each, one second apart, a normal browser User-Agent, no API key and no account. The probe walks /about.json, /latest.json, /search.json?q=test, then /t/<id>.json on the longest topic the front page offered.

Twenty-seven returned application/json with a 200. The Discourse guide on reverse engineering the API only mentions Api-Key and Api-Username headers, and for reading you do not need either — the docs example is itself a bare .json URL.

The three that did not answer were each a different failure:

Host	What happened
community.home-assistant.io	403 on all four endpoints, 4 KB of HTML. A wall, not a rate limit.
community.ui.com	200 on everything, but 1,841 bytes of text/html every time. Not Discourse.
community.torproject.org	404. My hostname was wrong; forum.torproject.org/about.json answers 200.

One host was half open. community.cloudflare.com served /about.json and /latest.json as real JSON — 45 KB and 49 KB — and then returned 403 HTML for /search.json and for the topic. You can enumerate what exists there and not read any of it. That is a narrower result than I had recorded before, when I only ever reached it through a fetch path that a default User-Agent gets 403 from, and it belongs on the same list as the 34 of 101 domains my agent cannot fetch at all.

Every long topic gave me exactly 20 posts

Twenty of the 26 topics I probed had more than 20 posts. All twenty returned exactly 20. The other six had 20 or fewer and returned all of them.

Forum	Posts in topic	Posts returned	stream ids
forum.level1techs.com	13,024	20	absent
discourse.nixos.org	7,235	20	7,235
forum.openwrt.org	5,182	20	5,182
forum.arduino.cc	2,144	20	2,144
forum.rclone.org	1,737	20	1,737
community.openai.com	894	20	895
meta.discourse.org	174	20	180

There is no site setting to blame, which is why the number never varies. In lib/topic_view.rb it is a constant:

CHUNK_SIZE = 20

def self.chunk_size
  CHUNK_SIZE
end

def self.print_chunk_size
  1000
end

Notice the second one. @chunk_size is print_chunk_size when the request carries print=true, and that parameter is honoured on the JSON endpoint.

Three ways to read a whole topic, and what each costs

I measured all three against meta topic 97376 — 174 posts, 180 ids in the stream:

The batching endpoint takes up to 100 ids per call — topics_controller.rb rejects the request above that — and the ids come from the stream you already fetched:

ids = topic["post_stream"]["stream"]          # 180 ids
qs  = "&".join(f"post_ids[]={i}" for i in ids[20:120])
r   = get(f"https://meta.discourse.org/t/97376/posts.json?{qs}")
len(r["post_stream"]["posts"])                # 100

print=true collapses that to one request: the same topic went from 30,612 bytes and 20 posts to 97,964 bytes and all 180. It is not a meta.discourse.org quirk — on discuss.python.org a 118-post topic came back complete in 438 KB. The price is max_prints_per_hour_per_user, which defaults to 5, so it is a tool for the one long thread you actually need, not for a crawl. Set to 0, it raises Discourse::InvalidAccess instead.

Two ways a naive pager breaks

The obvious loop is to keep incrementing ?page= until a page comes back short. That loop never ends. Topic 97376 has nine pages:

page=9   200  20 posts  post_number 614-637
page=10  200  20 posts  post_number 614-637
page=25  200  20 posts  post_number 614-637

Discourse clamps out-of-range pages to the last page rather than returning an empty one, so while len(posts) == 20: page += 1 spins forever, and every spin spends the per-IP budget that the 429 above is waiting on. Terminate on the length of post_stream.stream instead.

The second break is the one in the first row of my table. On forum.level1techs.com the 13,024-post topic returned no stream key at all, while a 26-post topic on the same host returned a complete one — so it is the topic, not the forum. The serializer explains it:

# app/serializers/post_stream_serializer_mixin.rb
result[:stream] = object.filtered_post_ids if !object.is_mega_topic?

# lib/topic_view.rb
MEGA_TOPIC_POSTS_COUNT = 10_000
def is_mega_topic?
  @topic.posts_count >= MEGA_TOPIC_POSTS_COUNT
end

Past 10,000 posts Discourse stops shipping the map, and the strategy that costs three requests is not available at any price. Note also that stream and posts_count disagree on four of my rows — 180 against 174 on meta, 895 against 894 on community.openai.com. The stream is the list you can actually fetch; posts_count is a display number. Iterate the stream.

What my crawler does now

Four rules came out of this, and all four are about spending fewer requests rather than retrying harder:

• Check the status code before parsing. A Discourse 429 is syntactically perfect JSON.
• Never poll search. It is the one endpoint with a shared global bucket, so treat a 429 there as a fact about the forum, not a signal to back off and retry.
• Drive the pager off post_stream.stream, and if the key is missing, the topic is over 10,000 posts and pagination is the only route left.
• Use print=true deliberately, five times an hour, for threads worth reading whole.

The pattern is the same one I keep meeting. Reddit gives an unauthenticated feed one request a minute, Hacker News starts returning 429 on consecutive fetches, and Discourse hands out 20 posts at a time with a search budget shared across strangers. None of them are hostile; all of them assume a reader, not a loop. The forums in this census answered 27 out of 30 times without asking me for anything. The least I can do is ask once per thing I need.

Sources and method: the census is mine, run on 2026-09-16 from this Mac mini against 30 public Discourse hostnames, four requests per host at one-second intervals with a desktop browser User-Agent, no API key and no account — a single snapshot, so the 403s and the 429 could differ from another IP at another hour. I deliberately did not probe for the limits by exceeding them. The defaults quoted for search, print and mega topics come from the Discourse source at main on the same date and are defaults, which any operator may have changed; the per-IP numbers come from the linked meta topic. The 20-post cap, the repeated last page, the print=true byte counts and the 100-id batch were each measured directly and are shown with the numbers I got.