Humanitarian and development organisations love to collect and store vast quantities of data but finding what they need is a challenge; is it?
Humanitarian and development organisations love to collect and store vast quantities of data. A colleague of mine, innovation manager at a European NGO, was recently complaining about how difficult and slow it is to search a database of more than 100,000 beneficiaries. The same week a UN partner was telling me how much they struggle to search for relevant pdf and excel documents. I told both of them that there is a simple solution!
Search is the new navigation
Searching for relevant information has become the leading paradigm of the web. Google became one of the biggest companies in the world just by providing a simple yet very efficient search bar.
Search is the new way to interact with platforms and systems, and users have grown to expect a simple search bar that will find the relevant information, document, report or graph in less than a second.
In one of my recent posts on bots I discussed data access as one of the main challenges of today’s data management. Some very innovative tools exist to better access data, such as virtual assistants, but I realised that, in most cases, organisations do not have a proper search tool in place to access their internal documents. Google is becoming the gateway to all web searches (although there are many other tools you can use that respect your privacy, like Qwant. Just saying…). But what about internal data and documents, excel files, databases and social media feeds?
There is a simple solution, and it’s free
The solution exists, is simple to implement, and is free and open source – it’s called Elasticsearch.
Wikipedia is using it to search full-text to provide suggested text; the Guardian deployed a solution based on Elasticsearch to give editors live feedback on public opinion about published articles through social media and visitor data; Airbus uses it to search its technical data, which is composed of more than 2 billion documents (yes billion with a b!); and UNEP has developed a cross-lingual search platform powered by Elasticsearch that tracks global communication flows to reveal social perceptions of environmental issues and help visualise the public debate around the UN’s Sustainable Development Goals.
5 reasons you should Elasticsearch
1. It is fast!
Based on a relatively new type of technology, Elasticsearch is a really, really fast way to search for almost anything, and especially text. The tool is known for searching millions of documents in a few milliseconds. Let’s not go into detail saying that NoSQL and word-level inverted indices are awesome ideas, but just remember that it is super fast!
The number of data sources we have is ever-increasing. We constantly add new datasets, databases, documents and libraries. It is a constant challenge to keep our applications performing well. Elasticsearch is designed to be scaled up by adding new “nodes”.
3. It is really fast!
Did I mention that it is fast?
4. Ease of use
There are many ways to use this tool. You can literally go on their website, download the tool and start using it (let’s be fair, some IT background is needed). Or ask you IT department to deploy it for you. Elastic search comes with a wonderful visualization tool called Kibana; and we all love Dashboard; don’t we…
It’s open source and free to use! What could be better? To implement it efficiently you might need some tech expertise, but there are lots of wonderful data experts out there that can help you with that, including us 😉