Library agnostic in-process recording of http(s) requests and responses

Overview

@gr2m/http-recorder

Library agnostic in-process recording of http(s) requests and responses

Test

Install

npm install @gr2m/http-recorder

Usage

import http from "node:http";
import httpRecorder from "@gr2m/http-recorder";

httpRecorder.start();
httpRecorder.addListener(
  "record",
  ({ request, response, requestBody, responseBody }) => {
    const { method, protocol, host, path } = request;
    const requestHeaders = request.getHeaders();

    console.log(`> %s %s//%s%s`, method, protocol, host, path);
    console.log(`> %j`, requestHeaders);
    console.log(Buffer.concat(requestBody).toString());

    const { statusCode, statusMessage, headers: responseHeaders } = response;
    console.log(`\n< %s %s`, statusCode, statusMessage);
    console.log(`< %j`, responseHeaders);
    console.log(Buffer.concat(responseBody).toString());
  }
);

const request = http.request("http://httpbin.org/post", { method: "post" });
request.write("data");
request.end();

// > POST http://httpbin.org/post
// > {"host":"httpbin.org"}
// data
//
// < 200 OK
// < {"content-type":"application/json",...}
// {
//   "args": {},
//   "data": "data",
//   ...
// }

API

httpRecorder is a singleton API.

httpRecorder.start()

Hooks into the request life cycle and emits record events for each request sent through the http or https modules.

httpRecorder.stop()

Removes the hooks. No record events will be emitted.

httpRecorder.addListener("record", listener)

Subscribe to a record event. The listener callback is called with an options object

httpRecorder.removeListener("record", listener)

Remove a record event listener.

httpRecorder.removeAllListeners()

Removes all record event listeners.

How it works

Once started, httpRecorder hooks itself into the http.ClientRequest.prototype.onSocket method which is conveniently called synchronously in the http.ClientRequest constructor.

When a request is intercepted, we

  1. hook into the request.write method and the request.end method in order to clone the request body
  2. subscribe to the response event
  3. hook into the response.emit method in order to clone the response body without consuming it

and then emit a record event with the request, response, requestBody and responseBody options.

Contributing

See CONTRIBUTING.md

Credit

The inspiration for hooking into http.ClientRequest.prototype.onSocket method comes from Mitm.js - an http mocking library for TCP connections and http(s) requests.

License

MIT

Comments
  • test delayed response without calling `.pause()`

    test delayed response without calling `.pause()`

    For example, I have a feeling the 693fc94 test case isn't exactly right. You're calling response.pause in the user's code, but that's not necessary. As a user you can do the following and get the expected output:

    var Http = require("http")
    
    Http.get("http://example.com", function(res) {
    	setTimeout(function() {
    		res.on("data", (data) => console.log("Data:", data.toString()))
    	}, 500)
    })
    

    The hook into response's data event in the recording code is the thing that changes behavior โ€” it immediately unpauses the stream. I suppose you could call res.pause in the recording code, but I have a suspicion that may then break automatic unpausing on data the user code does. โ€”https://github.com/gr2m/http-recorder/issues/3#issuecomment-999501190

    released 
    opened by gr2m 8
  • Add more tests

    Add more tests

    Follow up to https://github.com/moll/node-mitm/issues/76#issuecomment-998713363 thanks @moll

    • [x] #4
    • [x] #5
    • [x] Does piping one stream into OutgoingMessage work, too? ๐Ÿ‘‰ #2
    • [x] #11
    • [x] #9
    opened by gr2m 5
  • hangs when recording `got.stream.post`

    hangs when recording `got.stream.post`

    See example at https://github.com/sindresorhus/got/blob/main/documentation/3-streams.md

    import fs from "node:fs";
    import fs from "node:fs";
    import { promisify } from "node:util";
    
    import HttpRecorder from "@gr2m/http-recorder";
    import got from "got";
    
    const pipeline = promisify(stream.pipeline);
    
    HttpRecorder.enable()
    
    // For POST, PUT, PATCH, and DELETE methods, `got.stream` returns a `stream.Writable`.
    pipeline(
      fs.createReadStream("test.txt"),
      got.stream.post("http:/httpbin.org/post"),
      process.stdout
    );
    
    released 
    opened by gr2m 2
  • rename `.enable()` to `.start()` and `.disable()` to `.stop()` as it works better with a

    rename `.enable()` to `.start()` and `.disable()` to `.stop()` as it works better with a "recorder"

    Breaking changes

    • rename .enable() to .start() and .disable() to .stop() as it works better with a "recorder"
    • prefer .addListener/.removeListener over .on/off for distinction from .start/.stop methods and web compat.
    released 
    opened by gr2m 1
  • Add test for calling `.{write,end}(buffer, callback)`

    Add test for calling `.{write,end}(buffer, callback)`

    Did you test when calling write, end et al. with a chunk and a callback

    โ€”https://github.com/gr2m/http-recorder/issues/3#issuecomment-999501190

    That's a minor thing, but given readable.setEncoding, the recorder is converting strings back to buffers presuming UTF-8(?).

    My thinking here was to normalize the requestBodyChunks passed to the the record event listeners. I'm not worried about the performance overhead as this is a dev tool. Do you see any other problem? I will default encoding to utf-8 as part of #17

    released 
    opened by gr2m 1
  • delayed response read

    delayed response read

    Related to missing output, I have a sense hooking into data immediately will cause some applications to miss parts of the content body. Those that don't start reading the response right after response and instead expect the stream to be paused for a while. Perhaps they're passing the response first to an async function, but by that time, the recording logic will have already unpaused the stream. โ€”https://github.com/moll/node-mitm/issues/76#issuecomment-998713363

    released 
    opened by gr2m 1
  • Hook into `OutgoingMessage.prototype.end`

    Hook into `OutgoingMessage.prototype.end`

    (the parent of ClientRequest) that according to https://github.com/nodejs/node/blob/113133badbe3c2f40f61eac733006207e06b9cc2/lib/_http_outgoing.js#L833 seems to call an internal method directly (not going through write). Wouldn't that miss output? โ€”https://github.com/moll/node-mitm/issues/76#issuecomment-998713363

    released 
    opened by gr2m 1
  • chore(deps): update dependency @types/node to v18

    chore(deps): update dependency @types/node to v18

    Mend Renovate

    This PR contains the following updates:

    | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | @types/node (source) | ^17.0.2 -> ^18.0.0 | age | adoption | passing | confidence |


    Configuration

    ๐Ÿ“… Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    ๐Ÿšฆ Automerge: Disabled by config. Please merge this manually once you are satisfied.

    โ™ป Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    ๐Ÿ”• Ignore: Close this PR and you won't be reminded about this update again.


    • [ ] If you want to rebase/retry this PR, click this checkbox.

    This PR has been generated by Mend Renovate. View repository job log here.

    opened by renovate[bot] 0
  • chore(deps): update dependency tsd to ^0.25.0

    chore(deps): update dependency tsd to ^0.25.0

    Mend Renovate

    This PR contains the following updates:

    | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | tsd | ^0.24.0 -> ^0.25.0 | age | adoption | passing | confidence |


    Release Notes

    SamVerschueren/tsd

    v0.25.0

    Compare Source

    • Upgrade to TypeScript 4.9 5a7ee7b

    v0.24.1

    Compare Source


    Configuration

    ๐Ÿ“… Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    ๐Ÿšฆ Automerge: Disabled by config. Please merge this manually once you are satisfied.

    โ™ป Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    ๐Ÿ”• Ignore: Close this PR and you won't be reminded about this update again.


    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

    opened by renovate[bot] 0
  • Event not emitted with all errors

    Event not emitted with all errors

    Certain http errors are not causing the "record" event to be emitted, making my http recorder logic not run. For example, if I call a url that doesn't exist (and I would imagine timeouts). I would like to be able to record events like this for testing purposes

    I believe this is because you are hooking into the node "response" event, which is probably not fired for errors in which there is no response, like timeouts. What do you think?

    opened by hmorey3 1
  • feat: Add response timing

    feat: Add response timing

    I'm working on a library which records the timing of HTTP requests in a test suite to generate metrics around slow HTTP requests, your library works great for intercepting these HTTP requests, but I can't see any easy way to record how long each request is taking.

    Here's a very short change which adds an extra attribute to the record event with the time taken for the request. I'm not particularlly tied to this implemention and so another option might be to add a "start" event, which would allow the timing of the requests outside of the library.

    If you're happy with the change then i'm ok if you add changes directly to the PR or let me know any cahnges you'd like.

    opened by mariovisic 1
Releases(v2.0.2)
Owner
Gregor Martynus
Community Engineer ๐Ÿ”ง๐Ÿค“๐ŸŒˆโœจ
Gregor Martynus
make streaming http requests

hyperquest treat http requests as a streaming transport The hyperquest api is a subset of request. This module works in the browser with browserify. r

James Halliday 711 Sep 8, 2022
Wrap native HTTP requests with RFC compliant cache support

cacheable-request Wrap native HTTP requests with RFC compliant cache support RFC 7234 compliant HTTP caching for native Node.js HTTP/HTTPS requests. C

Luke Childs 259 Dec 20, 2022
Node.js web server framework for Http/1.1 or Http/2

Node.js web server framework for Http/1.1 or Http/2 Description: This is http framework, you can use it to create Http/1.1 or Http/2 serviceใ€‚ Now let'

Jeremy Yu 10 Mar 24, 2022
A server side agnostic way to stream JavaScript.

JS in JSN A server side agnostic way to stream JavaScript, ideal for: inline hydration network free ad-hoc dependencies bootstrap on demand libraries

Andrea Giammarchi 26 Nov 21, 2022
HTTP server mocking and expectations library for Node.js

Nock HTTP server mocking and expectations library for Node.js Nock can be used to test modules that perform HTTP requests in isolation. For instance,

Nock 11.9k Jan 3, 2023
๐ŸŒ Human-friendly and powerful HTTP request library for Node.js

Sindre's open source work is supported by the community. Special thanks to: Human-friendly and powerful HTTP request library for Node.js Moving from R

Sindre Sorhus 12.5k Jan 9, 2023
Promise based HTTP client for the browser and node.js

axios Promise based HTTP client for the browser and node.js New axios docs website: click here Table of Contents Features Browser Support Installing E

axios 98k Dec 31, 2022
Ajax for Node.js and browsers (JS HTTP client)

superagent Small progressive client-side HTTP request library, and Node.js module with the same API, supporting many high-level HTTP client features T

Sloth 16.2k Jan 1, 2023
Full-featured, middleware-oriented, programmatic HTTP and WebSocket proxy for node.js

rocky A multipurpose, full-featured, middleware-oriented and hackable HTTP/S and WebSocket proxy with powerful built-in features such as versatile rou

Tom 370 Nov 24, 2022
Very very very powerful, extensible http client for both node.js and browser.

ES-Fetch-API ไธญๆ–‡ | English Very very very powerful, extensible http client for both node.js and browser. Why should you use ES-Fetch API? Still using a

null 17 Dec 12, 2022
๐ŸŠ๐Ÿพ Simplified HTTP request client.

Deprecated! As of Feb 11th 2020, request is fully deprecated. No new changes are expected to land. In fact, none have landed for some time. For more i

request 25.6k Jan 4, 2023
A full-featured http proxy for node.js

node-http-proxy node-http-proxy is an HTTP programmable proxying library that supports websockets. It is suitable for implementing components such as

http ... PARTY! 13.1k Jan 3, 2023
HTTP Client Utilities

@hapi/wreck HTTP client utilities. wreck is part of the hapi ecosystem and was designed to work seamlessly with the hapi web framework and its other c

hapi.js 383 Nov 1, 2022
Simplifies node HTTP request making.

Requestify - Simplifies node HTTP request making. Requestify is a super easy to use and extendable HTTP client for nodeJS + it supports cache (-:. Ins

Ran Mizrahi 222 Nov 28, 2022
Run HTTP over UDP with Node.js

nodejs-httpp - Run HTTP over UDP based transport and Bring Web in Peer or P2P styles main js modules: udt.js, httpp.js, udts.js and httpps.js, that's

AppNet.Link 142 Aug 2, 2022
Global HTTP/HTTPS proxy agent configurable using environment variables.

global-agent Global HTTP/HTTPS proxy configurable using environment variables. Usage Setup proxy using global-agent/bootstrap Setup proxy using bootst

Gajus Kuizinas 267 Dec 20, 2022
An HTTP Web Server for Chrome (chrome.sockets API)

An HTTP Web Server for Chrome (chrome.sockets API)

Kyle Graehl 1.2k Dec 31, 2022
HTTP Client for Visual Studio Code to POST JSON, XML, image, ... files to REST APIs

friflo POST Goal Main goal of this extension is storing all HTTP request & response data automatically as files in a VSCode workspace. This ensures th

Ullrich Praetz 2 Nov 18, 2021
ZippyDamn-lib! is a ZippyShare Unofficial library (downloader & search) for nodejs

ZippyDamn-lib! ZippyDamn-lib! is a ZippyShare Unofficial library (downloader & search) for nodejs Looking for a CLI? Try Zippydamn! CLI Installation U

Virdio Samuel 4 Oct 2, 2021