A fundamental problem with the existing API when searching is that you don't get a total result count back. This is fine for auto-complete-style applications (where you're rarely going to be showing more than a handful of results), but makes providing paginated results more difficult.

This changes the SearchEngine.search API slightly, to drop the previous start & limit arguments, as well as returning a Results object instead of just a sliced array.

The new Results object provides an iterator, the total results count, specific offset access, and slicing (where the dropped start & limit can be re-applied) as it's API. This makes dealing with larger result sets more palatable, and offers the ability to provide the user better information.

Unfortunately, it's backward-incompatible. Sucks to have to do a major revision bump so soon, & poor planning on my part.

Resolves #8.

Previously, this package was known as minisearch. However, there's already a long-standing package on NPM with the same name. And we're currently smaller than 500 lines (w/ docs!), so nanosearch it is.

I didn't look before naming the repository, but there's already a MiniSearch out there (https://www.npmjs.com/package/minisearch).

To avoid confusion, this should be renamed before the initial release.

Resolves #1, resolves #2, resolves #3 .

NOT BACKWARD COMPATIBLE! This should be the last backward-incompatible change before the proper 1.0.0 release.

This paves the way for better scoring, highlighting, etc. In the process of this work, we also gain better Unicode support, as well as index versioning & compatibility checks.

TODO:

• [x] Added versioning to the index
• [x] Rewrite the preprocessor process code, iterating over the document body & generating a list instead of effectively just .toLowerCase().replace().split()
• [x] Change the preprocessor process api to emit a list of [[word, position], [word, position], ...] values
• [ ] ~Change the tokenizer tokenize api to include the position as well~ Unneeded
• [x] Change the index to store lists of positions instead of just the count
• [x] Change scoring to check the length of the lists instead of just the count

I cheated out on this due to time (there's only so much you can do in a single lunch hour!), so there's really only some integration-y tests right now. Actual unit tests for each method of the library would be good.

A big deficiency with the existing code is that we're currently only storing the number of time a term appears in a document.

Ideally, we'd be storing a list of the positions of the terms instead. This would allow for better scoring (e.g. "how close are the terms") & better querying (e.g. exact matches).

Unfortunately, this is non-trivial to implement (& backward-incompatible):

• [ ] Rewrite the preprocessor process code, iterating over the document body & generating a list instead of effectively just .toLowerCase().replace().split()
• [ ] Change the preprocessor process api to emit a list of [[word, position], [word, position], ...] values
• [ ] Change the tokenizer tokenize api to include the position as well
• [ ] Change the index to store lists of positions instead of just the count
• [ ] Change scoring to check the length of the lists instead of just the count

This is more experimental than something to directly merge.

Using the complete works of Shakespeare from Folger, this adds an in-browser benchmark. This comprises a 5.2Mb corpus as plain text.

The results on a recent MacBook look like:

Starting benchmark...
Indexing...
Done indexing 42 documents.
Elapsed: 1.018 seconds

Searching...
Searching on "what light"...
Found 42 results.
Top 3 Results: lucrece_TXT_FolgerShakespeare.txt, shakespeares-sonnets_TXT_FolgerShakespeare.txt, romeo-and-juliet_TXT_FolgerShakespeare.txt

Searching on "romeo"...
Found 42 results.
Top 3 Results: romeo-and-juliet_TXT_FolgerShakespeare.txt, the-comedy-of-errors_TXT_FolgerShakespeare.txt, titus-andronicus_TXT_FolgerShakespeare.txt

Searching on "hamlet"...
Found 42 results.
Top 3 Results: hamlet_TXT_FolgerShakespeare.txt, romeo-and-juliet_TXT_FolgerShakespeare.txt, the-merchant-of-venice_TXT_FolgerShakespeare.txt

Searching on "denmark"...
Found 42 results.
Top 3 Results: hamlet_TXT_FolgerShakespeare.txt, titus-andronicus_TXT_FolgerShakespeare.txt, pericles_TXT_FolgerShakespeare.txt

Searching on "the fool"...
Found 42 results.
Top 3 Results: the-phoenix-and-turtle_TXT_FolgerShakespeare.txt, henry-v_TXT_FolgerShakespeare.txt, twelfth-night_TXT_FolgerShakespeare.txt

Searching on "whether tis nobler to suffer"...
Found 42 results.
Top 3 Results: the-phoenix-and-turtle_TXT_FolgerShakespeare.txt, macbeth_TXT_FolgerShakespeare.txt, venus-and-adonis_TXT_FolgerShakespeare.txt

Searching on "shepherdess"...
Found 42 results.
Top 3 Results: the-winters-tale_TXT_FolgerShakespeare.txt, the-comedy-of-errors_TXT_FolgerShakespeare.txt, venus-and-adonis_TXT_FolgerShakespeare.txt

Done searching.
Elapsed: 0.003 seconds (for all queries)


Total terms: 5836
Total terms starting with "a": 370
First 10 "a" terms: ado, abo, ake, are, arb, ara, a, and, aul, ael
Index size (as JSON): 18.8 Mb

Searches are returning all documents (due to a lack of stop words & indexing some tiny/common n-grams). Regardless, performance is very decent.

Being able to use the index between page refreshes would be ideal. Some kind of local storage within the user's browser.

LocalStorage is easy, but has size restrictions. IndexedDB lacks those size restrictions, but I've never poked at it.