HTTP/HTTPS client has no response-body framing (chunked / Content-Length / close-terminated)

[binaryduke]

The HTTP and HTTPS clients have no way to read a response body correctly, because there is no RFC-7230 message-framing logic anywhere in the library. Grepping the tree for Transfer-Encoding, chunked, Content-Length or any body-reading helper returns nothing (checked in V1.4.0 and current main / V1.5.0).

Symptom

Issuing a GET against any real-world server and trying to read the body via stream::await / resolve / pipe gives a truncated body, a hang, or the raw chunked framing bytes mixed into the payload — depending on how the server chose to frame the response. There is no correct way to consume the body today.

Why it can't be worked around cleanly at the call site

The caller cannot know which of the three framings the server used without parsing the headers and then applying the matching read strategy — which is precisely the logic that belongs in the client.

What's needed

A read_body() on the http/https client implementing the standard three cases:

1. Transfer-Encoding: chunked → read and de-chunk until the terminating zero-length chunk
2. Content-Length: N → read exactly N bytes
3. neither → read until the server closes the connection (close-terminated)

We have implemented exactly this as an additive http::read_body() / https::read_body() (~186 lines, no change to stream::await / resolve / pipe, so nothing existing breaks). Happy to open a PR if you want it.

Related, and arguably a regression

V1.5.0 appears to have removed the request-side Content-Length write that V1.4.0 had. If so, an outgoing POST body is now itself unframeable — the same class of bug in the other direction. Worth a look.

[binaryduke]

Following up on this one, since #32 moved so quickly — we have this implemented and tested, and we're happy to open a PR.

What we have:

• http::read_body() / https::read_body() — additive, ~186 lines
• Implements the three RFC-7230 cases: Transfer-Encoding: chunked (de-chunk to the terminating zero-length chunk), Content-Length: N (read exactly N), and neither (read until the server closes)
• stream::await / resolve / pipe are untouched, so nothing existing changes behaviour
• Proven in production against a real-world scraping endpoint (chunked responses, ~20 documents parsed per call)

Two notes that may affect how you'd want it shaped:

1. We wrote it against V1.4.0. If you'd prefer it against current main (V1.5.0_1), say the word and we'll rebase before opening the PR.

2. V1.5.0 appears to have removed the request-side Content-Length write that V1.4.0 had. If that's right, an outgoing POST body is now unframeable too — the same class of bug in the other direction. Worth a look, and we'd be glad to include a fix for that in the same PR if you agree it's a regression.

Just let us know which branch you'd like it against.

[binaryduke]

Correction: our "V1.5.0 removed the request-side Content-Length write" claim was wrong.

We compared against our own patched working tree instead of a clean checkout, and misread our own patch's absence as a regression on your side. There was no regression. Sorry for the noise.

Restated correctly — Content-Length appears nowhere in include/ in either version:

git grep -in content-length 1.4.0 -- include/   # no output
git grep -in content-length 1.5.0 -- include/   # no output

So it's a long-standing gap present in both, not something 1.5.0 broke. Please don't spend any time hunting it.

Our patch closes it in both directions:

• request side — write Content-Length when a body is present, so an outgoing POST body is framed;
• response side — an RFC 7230 §3.3.3 body-length reader: Transfer-Encoding: chunked decoded properly (chunk-extensions and trailers, terminating on the explicit 0\r\n chunk); Content-Length: N read as an exact bounded count; and neither-present falling back to read-until-close with an idle deadline, so a peer that goes quiet without ever closing can't hang the caller.

All three drain the client's borrow buffer first, since read_header() / read_line() may already have pulled body bytes off the socket in the same underlying read as the headers — skipping that silently dropped body bytes in an earlier version of ours.

It is purely additive: stream::await() / resolve() / pipe() are untouched, so no existing caller changes behaviour. The new entry points are http::read_body() / https::read_body().

We've rebased it onto tag 1.5.0 — clean 3-way merge; the only conflict was your new HTTP_NODEPP namespace, which ours now simply extends (no symbol collisions).

Which branch would you like the PR against — main, or the new dev?

One heads-up before you take it: we can't currently run the WebSocket half of our suite on 1.5.0 at all — see #36. If you'd rather we target 1.4.0 until that's resolved, say the word and we'll rebase again.