📦
📦
🔥
npm-registry-firewall npm registry proxy with on-the-fly filtering
Key Features
- Restricts access to remote packages by predicate:
name
org
version
rangelicense
typedateRange
age
username
- custom
filter
function - vulnerability level via builtin npm-registry-firewall/audit plugin
- Flexible configuration: use presets, plugins and define as many
server/context-path/rules
combinations as you need. - Extendable. expressjs-inspired server implementation is under the hood.
- Standalone. No clouds, no subscriptions.
- Linux / Windows / macOS compatible.
- Has no deps. Literally zero.
Motivation
To mitigate security and legal risks
Open Source is essential for modern software development. According to various estimates, at least 60% of the resulting codebase is composed of open repositories, libraries and packages. And it keeps growing. Synopsys OSSRA 2021 report found that 98% of applications have open source dependencies.
But open does not mean free. The price is the risk that you take:
- Availability
- Security
- Legality / license
Let's consider these problems in the context of the JS universe.
Availability risks
JS packages are distributed in various ways: git repos, cdns and package registries. Regardless of the method, there are only two entry types that are finally resolved by any pkg manager: git-commit pointers and tarball links.
"dependencies": {
"yaf" : "git://github.com/antongolub/yarn-audit-fix.git#commit-hash",
"yaf2": "antongolub/yarn-audit-fix",
"yarn-audit-fix" : "*"
}
yaf2@antongolub/yarn-audit-fix:
version "9.2.1"
resolved "https://codeload.github.com/antongolub/yarn-audit-fix/tar.gz/706646bab3b4c7209596080127d90eab9a966be2"
dependencies:
"@types/find-cache-dir" "^3.2.1"
"@types/fs-extra" "^9.0.13"
"node_modules/yaf": {
"name": "yarn-audit-fix",
"version": "9.2.1",
"resolved": "git+ssh://[email protected]/antongolub/yarn-audit-fix.git#706646bab3b4c7209596080127d90eab9a966be2",
"license": "MIT",
"node_modules/yarn-audit-fix": {
"version": "9.2.1",
"resolved": "https://registry.npmjs.org/yarn-audit-fix/-/yarn-audit-fix-9.2.1.tgz",
"integrity": "sha512-4biFNP4ZLOHboB2cNVuhYyelTFR/twlfmGMQ2TgJgGRORMDM/rQdQqhJdVLuKvfdMLFEPJ832z6Ws5OoCnFcfA==",
"dependencies": {
So the implementation of mirroring is fundamentally quite simple: we just need to save and expose these assets from an alternative ssh/https entry point. Luckily this has already happened. The main repository for JS code is registry.npmjs.org. And at least 5 public replicas are always available as alternatives:
- https://registry.yarnpkg.com
- https://registry.npmmirror.com
- https://r.cnpmjs.org
- https://skimdb.npmjs.com/registry
- https://registry.npm.taobao.org
If this reliability level is not enough, you can easily run one more registry:
Security risks
Any code may not work properly. Due to error or malice. Keep in mind that most OSS licenses exclude any liability for damages. It's also important to always remember that oss code is not verified before being published. These two circumstances sometimes give rise to dangerous incidents like colors.js or node-ipc.
The independent audit process is expensive, time consuming, so only setting a delay before using new pkg version might be effective countermeasure.
Legal risks
License agreement is an attribute of the moment: it can suddenly change and affect the development process (for example, husky-5). Uncontrolled use of new versions may have legal and financial consequences. Therefore, automated license checks should be part of CI/CD pipeline or the registry's own feature.
Implementation notes
The proxy intercepts packuments and tarball requests and applies the specified filters to them:
- Removes all forbidden entries from packuments.
- Returns 404 if the filtered packument
versions
are empty. - Returns 404 if
registy/[email protected]
does not satisfy the policies. - Passes any other requests to the remote registry as is.
Requirements
Node.js >= 14
Install
# npm
npm i npm-registry-firewall
# yarn
yarn add npm-registry-firewall
Usage
CLI
npm-registry-firewall /path/to/config.json
JS API
import {createApp} from 'npm-registry-firewall'
const app = createApp({
server: {
host: 'localhost',
port: 3001,
},
firewall: {
registry: 'https://registry.npmmirror.com',
rules: [
{
policy: 'allow',
org: '@qiwi'
},
{
policy: 'deny',
name: '@babel/*,react@^17' // All @babel-scoped pkgs and react >= 17.0.0
},
{
policy: 'allow',
filter: ({name, org}) => org === '@types' || name === 'react' // may be async
},
{
plugin: [['npm-registry-firewall/audit', {
critical: 'deny',
moderate: 'warn'
}]]
},
]
}
})
await app.start()
TS libdefs
Included
type LetAsync<T> = T | Promise<T>
type TApp = {
start: () => Promise<void>
stop: () => Promise<void>
}
type TLogger = typeof console
type TServerConfig = {
host?: string
port?: string | number
base?: string
healthcheck?: string | null
metrics?: string | null
secure?: {
key: string,
cert: string
}
requestTimeout?: number
headersTimeout?: number
keepAliveTimeout?: number
extend?: string
}
type TPolicy = 'allow' | 'deny' | 'warn'
type TRule = {
policy?: TPolicy
name?: string | RegExp | Array<string | RegExp>
org?: string | RegExp | Array<string | RegExp>
dateRange?: [string, string]
age?: number | [number] | [number, number]
version?: string,
license?: string | RegExp | Array<string | RegExp>
username?: string | RegExp | Array<string | RegExp>
filter?: (entry: Record<string, any>) => LetAsync<boolean | undefined | null>
extend?: string
plugin?: TPluginConfig
}
type TPluginConfig = string | [string, any] | TPlugin | [TPlugin, any]
type TCacheConfig = {
ttl: number
evictionTimeout?: number
name?: string
}
type TCacheImpl = {
add(key: string, value: any, ttl?: number): LetAsync<any>
has(key: string): LetAsync<boolean>
get(key: string): LetAsync<any>
del(key: string): LetAsync<void>
}
type TCacheFactory = {
(opts: TCacheConfig): TCacheImpl
}
type TFirewallConfig = {
registry: string
entrypoint?: string
token?: string
base?: string
rules?: TRule | TRule[]
cache?: TCacheConfig | TCacheImpl | TCacheFactory
extend?: string
}
type TConfig = {
server: TServerConfig | TServerConfig[]
firewall: TFirewallConfig
extend?: string
}
type TValidationContext = {
options: any,
rule: TRule,
entry: Record<string, any>
boundContext: {
logger: TLogger
registry: string
authorization?: string
entrypoint: string
name: string
org?: string
version?: string
}
}
type TPlugin = {
(context: TValidationContext): LetAsync<TPolicy>
}
type TAppOpts = {
logger?: TLogger
cache?: TCacheFactory
}
export function createApp(config: string | TConfig | TConfig[], opts?: TAppOpts): Promise<TApp>
export function createLogger(
extra?: Record<string, any>,
formatter?: (logCtx: {level: string, msgChunks: string[], extra: Record<string, any>}) => void
): string
Config
{
"server": {
"host": "localhost", // Defaults to 127.0.0.1
"port": 3000, // 8080 by default
"secure": { // Optional. If declared serves via https
"cert": "ssl/cert.pem",
"key": "ssl/key.pem"
},
"base": "/", // Optional. Defaults to '/'
"healthcheck": "/health", // Optional. Defaults to '/healthcheck'. Pass null to disable
"metrics": "/metrics", // Optional. Uptime, CPU and memory usage. Defaults to '/metrics'. null to disable
"keepAliveTimeout": 15000, // Optional. Defaults to 61000
"headersTimeout": 20000, // Optional. Defaults to 62000
"requestTimeout": 10000 // Optional. Defaults to 30000
},
"firewall": {
"registry": "https://registry.npmmirror.com", // Remote registry
"token": "NpmToken.*********-e0b2a8e5****", // Optional bearer token
"entrypoint": "https://r.qiwi.com/npm", // Optional. Defaults to `${server.secure ? 'https' : 'http'}://${server.host}:${server.port}${route.base}`
"base": "/", // Optional. Defaults to '/'
"cache": { // Optional. Defaults to no-cache (null)
"ttl": 5, // Time to live in minutes. Specifies how long resolved pkg directives will live.
"evictionTimeout": 1 // Cache invalidation period in minutes. Defaults to cache.ttl.
},
"extends": "@qiwi/internal-npm-registry-firewall-rules", // Optional. Populates the entry with the specified source contents (json/CJS module only)
"rules": [
{
"policy": "allow",
"org": "@qiwi"
},
{
"policy": "allow",
"name": ["@babel/*", "@jest/*", "lodash"] // string[] or "comma,separated,list". * works as .+ in regexp
},
{
"policy": "warn", // `warn` directive works like `allow`, but also logs if someone has requested a tarball matching the rule
"name": "reqresnext"
},
{
"policy": "deny",
"extends": "@qiwi/nrf-rule", // `extends` may be applied at any level, and should return a valid value for the current config section
},
{
"plugin": ["npm-registry-firewall/audit", {"moderate": "warn", "critical": "deny"}]
},
{
"policy": "deny",
"name": "colors",
"version": ">= v1.4.0" // Any semver range: https://github.com/npm/node-semver#ranges
},
{
"policy": "deny",
"license": "dbad" // Comma-separated license types or string[]
},
{
"policy": "allow",
"username": ["sindresorhus", "isaacs"] // Trusted npm authors.
},
{
"policy": "allow",
"name": "d",
// `allow` is upper, so it protects `< 1.0.0`-ranged versions that might be omitted on next steps
"version": "< 1.0.0"
},
{
"policy": "deny",
// Checks pkg version publish date against the range
"dateRange": ["2010-01-01T00:00:00.000Z", "2025-01-01T00:00:00.000Z"]
},
{
"policy": "allow",
"age": 5 // Check the package version is older than 5 days. Like quarantine
}
]
}
}
Multi-config
// Array at the top level
[
// Two servers (for example, http and https) share the same preset
{
"server": [
{"port": 3001},
{"port": 3002, "secure": {"cert": "ssl/cert.pem", "key": "ssl/key.pem" }},
],
"firewall": {
"registry": "https://registry.yarnpkg.com",
"rules": {"policy": "deny", "org": "@qiwi"}
}
},
// One server has a pair of separately configured endpoints
{
"server": {"port": 3003},
"firewall": [
{"base": "/foo", "registry": "https://registry.npmjs.org", "rules": {"policy": "deny", "org": "@qiwi"}},
{"base": "/bar", "registry": "https://registry.yarnpkg.com", "rules": {"policy": "deny", "org": "@babel"}}
]
}
]
Cache
By default, nrf uses a simple in-memory cache to store patched packuments.
cache: { // Optional. Defaults to no-cache (null)
ttl: 5, // Time to live in minutes. Specifies how long resolved pkg directives will live.
evictionTimeout: 1, // Cache invalidation period in minutes. Defaults to cache.ttl.
name: 'unique' // If and only if you use the same rules for several firewall entrypoints (multi-port proxy)
// you can slighly optimise resource consupmtion by sharing the cache. Defaults to `randId()`
}
You can also provide your own implementation instead, for example, to create cassandra-based distributed cache:
import {createApp} from 'npm-registry-firewall'
const cache = {
add() {}, // each method may be async
has() {return false},
get() {},
del() {}
}
const app = createApp({
server: {port: 5000},
firewall: {
registry: 'https://registry.npmjs.org',
cache,
rules: []
}
})
Or even a cache factory:
const cache = () => {
// ... init
return {
get() {},
...
}
}
Pass null
as config.firewall.cache
to disable.
Extras
Presets
Introduce your own reusable snippets via extends
or preset
. This statement can be applied at any config level and should return a valid value for the current section. The specified path will be loaded synchronously through require
, so it must be a JSON or CJS module.
const config = {
// should return `firewall` and `servers`
extends: '@qiwi/nrf-std-config',
server: {
port: 5000,
extends: '@qiwi/nrf-server-config'
},
firewall: {
// `rules`, `registry`, etc,
extends: '@qiwi/nrf-firewall-config',
// NOTE If you redefine `rules` the result will be contatenation of `[...rules, ...extends.rules]`
rules: [{
policy: 'deny',
// `name`, `org`, `filter`, etc
extends: '@qiwi/nrf-deprecated-pkg-list'
}, {
policy: 'allow',
extends: '@qiwi/nrf-whitelisted-orgs'
}, {
extends: '@qiwi/nrf-all-in-one-filter'
}]
}
}
For example, extends
as a filter:
// '@qiwi/nrf-all-in-one-filter'
module.exports = {
filter({org, name, time, ...restPkgData}) {
if (name === 'react') {
return true
}
if (org === '@babel') {
return false
}
if (restPkgData.license === 'dbad') {
return false
}
}
}
Plugins
Plugin is slightly different from preset:
- Async. It's loaded dynamically as a part of rule processing pipeline, so it may be an ESM.
- Configurable. Opts may be passed as the 2nd tuple arg.
- Composable. There may be more than one per
rule
.
const rule1 = {
plugin: ['@qiwi/nrf-plugin']
}
const rule2 = {
plugin: [
['@qiwi/nrf-plugin', {foo: 'bar'}],
'@qiwi/nrf-another-one'
]
}
Plugin interface is an (async) function that accepts TValidationContext
and returns policy type value or false
as a result:
const plugin = ({
rule,
entry,
options,
boundContext
}) => entry.name === options.name ? 'deny' : 'allow'
npm-registry-firewall/audit
Some registries do not provide audit API, that's why the plugin is disabled by default. To activate, add a rule:
{
plugin: [['npm-registry-firewall/audit', {
critical: 'deny',
moderate: 'warn'
}]]
}
Monitoring
/healthcheck
{"status":"OK"}
/metrics
{
"uptime": "00:00:47",
"memory": {
"rss": 34320384,
"heapTotal": 6979584,
"heapUsed": 5632224,
"external": 855222,
"arrayBuffers": 24758
},
"cpu": {
"user": 206715,
"system": 51532
}
}
stdout
{"level":"INFO","timestamp":"2022-04-11T20:56:47.031Z","message":"npm-registry-firewall is ready for connections: https://localhost:3000"}
{"level":"INFO","timestamp":"2022-04-11T20:56:49.568Z","traceId":"44f21c050d8c6","clientIp":"127.0.0.1","message":"GET /d"}
{"level":"INFO","timestamp":"2022-04-11T20:56:50.015Z","traceId":"44f21c050d8c6","clientIp":"127.0.0.1","message":"HTTP 200 446ms"}
logger
You can override the default implementation if needed:
import { createLogger, createApp } from 'npm-registry-firewall'
const logger = createLogger(
{foo: 'bar'}, // extra to mix
({level, msgChunks, extra}) => JSON.stringify({
msg: msgChunks.map(m => '' + m),
mdc_trace: {spanId: extra.traceId, traceId: extra.traceId, bar: extra.foo},
timestamp: new Date().toISOString(),
level
})
)
const app = createApp(cfg, {logger})
Manual testing
.npmrc
registry=https://localhost:3000
strict-ssl=false
run
# node src/main/js/cli.js config.json
yarn start
npm view
npm-registry-firewall % npm view d versions
[ '0.1.0', '0.1.1' ]
curl
curl -k https://localhost:3000/registry/minimist/-/minimist-1.2.6.tgz > minimist.tgz
curl -k https://localhost:3000/registry/react > react.json
Contributing
Feel free to open any issues: bug reports, feature requests or questions. You're always welcome to suggest a PR. Just fork this repo, write some code, put some tests and push your changes. Any feedback is appreciated.