Hi, I know that you have optimized your library a lot.

But is there any chance to make the decompression (inflate) faster? I use it for decoding PNG images with UPNG.js (which calls pako.js) . For large PNG files, it takes quite a lot of time (about 50% of time takes PNG filtering and 50% takes Inflate).

I attach one of my PNG images. https://user-images.githubusercontent.com/15955558/28381646-e8566a36-6cbb-11e7-8d32-0d0f6ffe08a8.png

This is just an initial commit for you to see the approach before I spend time in something that you don't like, @puzrin .

With the new change, Rollup produces 3 bundles:

• CommonJS
• UMD
• ES6 module

package.json now exposes those 3 packages so most bundlers and libraries can choose to pull the better version.

This removes the need for browserify and rollup should be much more lightweight.

I can easily produce also 3 minimized bundles. Also, I wasn't sure why Inflate and deflate where packaged independently. But I can also add those 2 with the 3 modules.

This also open the door to use ES6 and more modern JS features and have rollup to produce a ES5 bundle using babel for example.

Wdyt??

Closes #97

In my case, I want to unzip some gzipped strings. What are the minimum steps I need to do this client side / browser?

I don't understand which files I need. I would expect justdist/pako_deflate.min.js but it requires files not in the dist directory.

Please provide a minimal example. Cooler would be examples for decode too, in the README.MD, but that's just thinking about other people who stumble across this interesting module - not for me.

ps I am using requirejs if that makes things easier.

I saw what was mentioned in #135 and #136 and decided to try out the performance benchmarks for myself.

Selected samples: (1 of 1)
 > lorem_1mb

Sample: lorem_1mb.txt (1000205 bytes raw / ~257018 bytes compressed)
 > deflate-fflate x 29.27 ops/sec ±1.97% (52 runs sampled)
 > deflate-fflate-string x 25.85 ops/sec ±2.58% (47 runs sampled)
 > deflate-imaya x 7.41 ops/sec ±2.72% (22 runs sampled)
 > deflate-pako x 16.39 ops/sec ±1.44% (44 runs sampled)
 > deflate-pako-string x 15.05 ops/sec ±0.50% (41 runs sampled)
 > deflate-pako-untyped x 11.53 ops/sec ±1.06% (31 runs sampled)
 > deflate-uzip x 29.20 ops/sec ±2.41% (51 runs sampled)
 > deflate-zlib x 25.09 ops/sec ±0.41% (45 runs sampled)
 > inflate-fflate x 252 ops/sec ±1.07% (87 runs sampled)
 > inflate-fflate-string x 141 ops/sec ±0.75% (78 runs sampled)
 > inflate-imaya x 159 ops/sec ±1.37% (83 runs sampled)
 > inflate-pako x 193 ops/sec ±0.32% (87 runs sampled)
 > inflate-pako-string x 59.54 ops/sec ±0.31% (61 runs sampled)
 > inflate-pako-untyped x 65.32 ops/sec ±1.22% (64 runs sampled)
 > inflate-uzip x 274 ops/sec ±0.73% (86 runs sampled)
 > inflate-zlib x 589 ops/sec ±0.29% (93 runs sampled)

If you would like to experiment with the patches I made, I forked the repo: 101arrowz/pako.

Note the addition of fflate and uzip. uzip was mentioned in the previous issues but is, as you mentioned, slightly unstable. It hangs on bad input rather than throwing an error, cannot be configured much, cannot be used asynchronously or in parallel (multiple calls to inflate in separate callbacks, for example, causes it to fail), and, worst of all, does not support streaming.

I created fflate to resolve these issues while implementing some of the popular features requested here (e.g. ES6 modules). In addition, fflate adds asynchronous support with Web and Node workers (offload to a separate thread entirely rather than just add to event loop) and ZIP support that is parallelized to easily beat out JSZip. The bundle size ends up much smaller than pako too because of the much simpler code. As you can see it is much faster than pako.

The purpose of this issue is not to boast about my library or to call pako a bad one. I believe pako can become much faster while still being more similar than fflate or uzip to the real zlib (i.e. the internal lib/ directory). The overhauled implementation in Node 12 as mentioned in #193 is the perfect chance to escape the trap of trying to match the C source nearly line-for-line and instead become a canonical but high-performance rework in JavaScript. For this reason, I suggest you try to optimize some of the slower parts of the library. I found an even better hashing function that produced very few collisions in my testing, and for larger files/image files it outperforms even Node.js zlib in C; if you'd like to discuss, please let me know.

Personally: thank you for your incredible contributions to the open source community. I use pako in multiple projects and greatly appreciate its stability, not present in libraries such as my own.

upd: https://github.com/nodeca/pako-test - bug's demo repo

Hi,

For one of my files during testing (it's a 93 MB heap dump file), I noticed that when the zlib.deflate is called the data size goes from 97186971 to 5772270, however if I immediately follow that with a call to zlib.inflate the result is 67615664 - so almost 30 megabytes of data seem to get lost somewhere. Is there some known issue of problems with particular files?

regards, Greg

I'm using the zstream interface directly instead of the inflate/deflate wrappers. Basically, I'm building a Node API compatible wrapper for use in browserify. If strm.next_out is set to a Node Buffer (or a browserify one) and typed arrays are present, utils.arraySet breaks. This is because the source buffer might be a typed array still so the test whether to use the typed array's subarray still passes, but the dest isn't a typed array so it breaks.

This just does a quick check to see if buffers are present and whether the dest is a buffer, and falls back to the normal loop copy if so.

Hello,

I'm currently using the following code with Pako 1.0.11:

pako.deflate(textEncode(source), { level: 9, to: 'string' })

I noticed that the option { to: 'string' } is not supported anymore on deflate (but still supported on inflate). The code to convert a buffer to a String is also still available at:

https://github.com/nodeca/pako/blob/45cce9f4d67dcd446e753a70ec31a56955ac20fc/lib/utils/strings.js#L99

So I'm wondering if this change is intended? And if by any chance we could restore this feature on deflate?

I can open a pull request with this change but first I want to make sure that we are in agreement :smile:

Thanks!

It seems deflate could produce different results for Chinese. e.g.: pako.deflate('中')

If we use pako.min.js, it would be, Uint8Array(11) [120, 156, 91, 187, 227, 9, 0, 4, 94, 2, 74]

If we use index.js(packed by webpack), it would b, Uint8Array(11) [120, 156, 123, 178, 99, 45, 0, 4, 204, 2, 74]

The latter would be correct, since it can be inflated to the original

Deflate is used in a PDF format for lossless compression.

A PDF file has been sent to me with a following sequence of 6419 bytes: "file.log".

When I try to inflate them with my own library UZIP.js, I get 71731 bytes: "file.txt".

Also, the PDF file can be displayed with pdf.js, and Adobe Reader (both seem to have their own Deflate implementations).

pako.js fails on this sequence of bytes, showing "Uncaught invalid distance too far back" file.log file.txt

This library looks like it supports node streams, but doesn't actually. It also makes it difficult to support streams because the last call to push has to contain the last chunk, when node streams usually end after emitting the last chunk.

This could be simplified by separating .push(chunk, false) and .push(chunk, true) into a .end(opt_chunk). Then it would be relatively easy to write a stream wrapper for this library.

Unless I'm just doing things wrong? Or there is a way to write an empty value to .push?

I would ideally want this to work:

var binaryCSV = require('binary-csv')
var createReadStream = require('filereader-stream');
createReadStream({ some gziped html5 file object })
  .pipe(pako.inflator())
  .pipe(binaryCSV())
  .on('data', function(line) { console.log('line', line) });

First of all, thank you very much for this great library!

I'm working on a project where i inflate very long strings and i wonder if you would consider adding support for TextEncoder and TextDecoder? This API is supported by all major browsers (only Edge is missing, but already under development) and you could use the current implementation as a fallback.

https://caniuse.com/#search=TextDecoder https://encoding.spec.whatwg.org/

I'm trying to uncompress a .zg file in chunks following the example in the documentation: https://nodeca.github.io/pako/#Inflate.prototype.push but run into error. Could you help me understand and resolve this error?

Browser console output

Response {type: 'basic', url: 'https:/****.rpt.gz', redirected: false, status: 200, ok: true, …} detail.js:48 ArrayBuffer(29235) detail.js:49 29235 detail.js:54 ArrayBuffer(8000) detail.js:54 ArrayBuffer(8000) detail.js:54 ArrayBuffer(8000) detail.js:54 ArrayBuffer(5235) detail.js:57 last chunk detail.js:64 something is wrong detail.js:78 Error: -3 at readReport (detail.js:65:10)

Source code

console.log(buf)
console.log(buf.byteLength)
const chunkSize = 8000
var inflate = new pako.Inflate({to: "string"})
for(let i = 0; i < buf.byteLength; i+= chunkSize){
    const chunk = buf.slice(i, i + chunkSize);
    console.log(chunk)
    if(i + chunkSize >= buf.byteLength){
	  inflate.push(chunk, true)
	  console.log("last chunk")
    }else{
	  inflate.push(chunk, false)
    }	
}
if(inflate.err){
    console.log("something is wrong")
    throw new Error(inflate.err)
}
console.log(inflate.result)

Note: I'm doing chunk because I get a "distance too far back error" which I understand means the file is too big to be inflated at once.

Looking forward to your reply, Charles

While developing a project in php, I've encountered the problem of a small number of demo projects using pako.js + php. So, I have a chance to create demo php + pako.js, implementing compression on client-side and decompression on server-side.

Here is repository: https://github.com/inplayo-com/pakojs-php-zlib-fetch

The author of pako.js please expand the list of demo projects, I think it will be very useful for community, appreciate this!

I have a particular file that can be read no problem with pako.js but if I try pako.min.js or pako_inflate.min.js it shows the error "incorrect header check" in the console.

pako_inflate_issues.zip

Seems to be an issue with your minimizer.

After the data has been sent to the push method with the Z_SYNC_FLUSH flag, no data appears in result. At the same time, Zlib node js does a good job on the same piece of data. Test code and file: test.zip I guess the problem is that zlib returns Z_OK instead of Z_STREAM_END.

Hello all,

I think I've found a bug in decompressing this data file. (It's an old data file that I did not create, and I had to do a lot of digging and guessing to figure out what the hell it was.) It appears to have a 2-byte zlib header (78 DA). From my reading through the docs and the code, it seems that by default pako.inflate() should auto-detect whether there's a zlib header as opposed to a gzip header. However, I get an incorrect header check error when doing pako.inflate().

const pako = require('pako');

let input = require('fs').readFileSync('KPAH_SDUS33_NVWVWX_200511060006');
let output = pako.inflate(input); // error: incorrect header check

The standard Python zlib library has no problems with it.

import zlib
with open('KPAH_SDUS33_NVWVWX_200511060006', 'rb') as finput:
    input = finput.read()
output = zlib.decompress(input)
print(len(output)) # 4000

For my specific application, I think I can get away with removing the zlib header and using pako.inflateRaw(), but it seems like pako.inflate() should be able to handle it.

Background: I'm trying to read a record from a git packfile using pako. This format embeds zlib data inside each record, and relies on zlib to indicate when the compressed data is done. In pako 1 this is easy enough - after the decompression is done I can use inflate.strm.avail_in to check how much input was consumed so I can read the next packfile record from the right spot. In pako 2, however, I no longer get a clean indication of the end of the compressed data, and instead get a -3 "incorrect header check" error.

Issue: When inflating a zlib stream with extra input data at the end (that's not another zlib stream), pako 2 gives me an "incorrect header check". This works fine in pako 1.

It looks like the inflate wrapper in pako 2 assumes the extra input data is another zlib stream and tries to start reading it again, which fails in this case because the extra data is unrelated. https://github.com/nodeca/pako/blob/0398fad238edc29df44f78e338cbcfd5ee2657d3/lib/inflate.js#L237-L245

Removing this from the wrapper allows it to work. Maybe this could be a configuration setting? Or changed to only happen for gzip streams? The gzip spec has repeated records, but zlib does not.

Small example:

const pako = require("pako");

const dataHex = 
    "789C0B492D2E5170492C49E4020013630345" // deflate("Test Data")
    + "14303893"; // 4 bytes of extra data

const data = new Uint8Array(dataHex.match(/.{1,2}/g).map(b => parseInt(b, 16))); // hex to Uint8Array

const inflate = new pako.Inflate({ windowBits: 15 });
inflate.push(data);

console.log("avail_in", inflate.strm.avail_in);
console.log("msg", inflate.msg);
console.log("result", new TextDecoder().decode(inflate.result));

Pako 1.0.11

avail_in: 4
msg: 
result: Test Data

Pako 2.*

avail_in: 2
msg: incorrect header check
result: