The issue to add Search to the Posts API, is literally the oldest open issue in this repository. It’s a specialist subject, so we’ve always hoped someone with specialist knowledge would come forward to help us solve it, but unfortunately that hasn’t happened.
To try to increase visibility of the problem we set up a page advertising contributor roles - detailing that we were looking for someone to help solve adding search to Ghost. That didn’t work either :(
Unfortunately, adding search is a particularly complex challenge. This is another attempt to try to get engagement from the wider OSS community on how to do this - we’d really appreciate the opinions and input of anyone who knows their search-fu.
There are two key use cases for search inside of Ghost itself - one being in the admin panel to find a post, tag, user or perhaps even a setting that you want to change. This would make a significant improvement to the usability the admin interface.
The other is adding search to the frontend of a blog, so that it becomes easier to find content. The main things that need to be searchable are post titles and content, tag names and description, user names, and probably meta titles and descriptions for the different resources. There are plenty of use cases where other things would also need to be searchable, so this will need to be extensible, but these are the key fields.
The best way to make both of these use cases possible, as well as making the search feature available to any apps or other extensions of Ghost in future, is to make it available through the API.
There are generally two approaches to solving search that would work in Ghost that I am aware of - the first being to use the FTS features of SQL and the second being to use a third party search tool of some description.
The upside to FTS is that, if we get it working well, it could work for all Ghost installs without any need to install additional (and likely complex) dependencies. The downside is that getting FTS to work for all three of sqlite, MySQL and PG is a sizeable challenge & the status of FTS in knex is unknown - but perhaps it can be done as a bookshelf/knex plugin, in a way that will benefit the wider community as well?
The upside to using a 3rd party tool, is that there are many modern and advanced systems that we could leverage to make an exceptional search feature. The downside is that most of them require additional complex dependencies* to be installed, which mean they aren’t viable options for Ghost core.
One of the tools that has been recommended is lunr.js, and there is a plugin for Ghost themes which combines lunr & the RSS feed to make search possible on the frontend. Supposedly lunr also works in node, not just the browser, but I’ve not found much information about it.
Other tools that are popular for search include lucene and elasticsearch and there are also numerous modules for node built on top of leveldb and redis databases. There are likely solutions out there that I haven’t heard about, so I’d be interested to hear ideas from others.
* complex dependencies includes binaries or external services - Ghost is currently installable on almost any platform without too much fiddling with compilers etc, and we want to keep it that way!
The ideal solution, in my mind, looks something like:
- Have a rudimentary version of search built into Ghost via the API - using FTS or something else - that will work for 90% of blogs (i.e. relatively small amounts of content).
- Implement this version of search to be easily extensible, so that it’s very easy to create an app that replaces the search with something more heavy-duty like elastic search or lucene, to support larger blogs.
The immediate future
As part of Zelda, we want to introduce a search bar to the admin interface which provides auto-complete style results for posts, tags and users. This should be possible using our existing API, some minor modifications (to allow us to fetch certain fields) and Ember. It’s not ideal, but I think something incredibly rudimentary is better than nothing at all! This will be spec’d in a separate issue.
Continuing this discussion
I’m really, really looking for the community to get involved and share their ideas here. This is a relatively specialist subject and I know there are tonnes of developers out there that have a great deal more experience implementing search than I do.
I’d like to hear thoughts, ideas and experiences on FTS - will we be able to make it work for all 3 databases without having to write too much custom code? Could a bookshelf plugin work? Is it an absolute nightmare not worth pursuing?
What about lunr? Is it a viable server-side option? Does it have too many requirements or not work for large data sets, or is it the perfect basic option provided it can easily be overridden with elastic search or something else?
Is there another solution that isn’t mentioned here? Remember we can’t have leveldb or redis or other complex dependencies in core and it has to work across sqlite3, mysql and postgres.
And what about the API? What would a good search API look like? Who out there has a great RESTful search API that we should take inspiration from?
If you know someone you think could answer any of these questions, please link them here and ask them to get involved!
server / core help wanted needs info