The Search Bottleneck in Traditional Databases
I once handled a project involving a product catalog with about 5 million records stored in PostgreSQL. Performance was excellent until the client requested a “Google-like” search bar. They needed fuzzy matching, relevance scoring, and instantaneous results across a dozen columns. While MySQL and PostgreSQL are powerhouses for structured data, they aren’t built for this specific type of heavy lifting.
SQL’s LIKE '%keyword%' operator is a performance trap. It forces the database to perform a full table scan, which is manageable at 10,000 rows but disastrous at 5 million. In our case, query times jumped from 50ms to nearly 4 seconds as the dataset grew. To solve this, you need an inverted index. For years, Elasticsearch was the standard, but licensing shifts pushed the community toward a truly open-source alternative: OpenSearch.
Why OpenSearch?
OpenSearch is a community-driven fork of Elasticsearch 7.10.2. It keeps the RESTful API you likely already know, making it a drop-in replacement for most existing apps. It excels at two core tasks: Full-text search for finding documents based on intent and Log Analytics for parsing millions of time-series events.
Think of it as the engine behind the OSD stack (OpenSearch, Fluentd/Logstash, and Dashboards). If your application is choking on text queries or you need to centralize logs from 50 different microservices, OpenSearch is the most reliable path forward. Let’s get it running on your machine using Docker.
System Prerequisites
Before pulling any images, you must adjust one critical system setting. OpenSearch relies on mmapfs to store its indices. Most Linux distributions set the default memory map limit too low, which causes the OpenSearch container to crash immediately with an “out of memory” error.
Run this command on your Linux host to increase the limit to the required 262,144:
sudo sysctl -w vm.max_map_count=262144To make this change survive a reboot, add vm.max_map_count=262144 to your /etc/sysctl.conf file. If you are on Docker Desktop for Windows or Mac, you may need to run this inside the underlying virtual machine.
Deploying with Docker Compose
Docker Compose is the cleanest way to manage the search engine and its web interface (Dashboards) together. Create a new project folder and drop in a docker-compose.yml file. We recommend having at least 4GB of RAM available on your host for a smooth experience.
The configuration below disables the security plugin. This is helpful for local development, but never do this in production. Always use SSL/TLS and strong authentication when exposing search data to the internet.
version: '3.8'
services:
opensearch-node:
image: opensearchproject/opensearch:2.11.0
container_name: opensearch-node
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node
- discovery.type=single-node
- bootstrap.memory_lock=true
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
- DISABLE_INSTALL_DEMO_CONFIG=true
- DISABLE_SECURITY_PLUGIN=true
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- osdata:/usr/share/opensearch/data
ports:
- 9200:9200
networks:
- opensearch-net
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:2.11.0
container_name: opensearch-dashboards
ports:
- 5601:5601
environment:
- 'OPENSEARCH_HOSTS=["http://opensearch-node:9200"]'
- "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true"
networks:
- opensearch-net
volumes:
osdata:
networks:
opensearch-net:Launch the stack with a single command:
docker-compose up -dGive the services about 30 to 60 seconds to warm up. You can monitor the progress by tailing the logs: docker logs -f opensearch-node.
Verifying the Installation
Check if the API is alive by hitting the default port. Use curl or any browser to visit the status endpoint.
curl http://localhost:9200You should see a JSON payload containing the version number (e.g., “2.11.0”) and the tagline “The OpenSearch Project: https://opensearch.org/”. If that appears, your engine is ready.
Now, navigate to http://localhost:5601 in your browser. This opens OpenSearch Dashboards. It’s a powerful GUI that lets you visualize data, manage your indices, and test queries without writing complex curl commands.
Indexing and Searching Data
OpenSearch uses different terminology than SQL. You don’t have “tables”; you have indices. You don’t have “rows”; you have documents. Let’s put this into practice.
1. Adding a Document
We’ll create an index called books and add one entry. OpenSearch is schema-less, so it will automatically detect that “publish_date” is a date and “title” is a string.
curl -X PUT "http://localhost:9200/books/_doc/1" -H 'Content-Type: application/json' -d'
{
"title": "The Phoenix Project",
"author": "Gene Kim",
"category": "IT & DevOps",
"summary": "A novel about IT, DevOps, and helping your business win.",
"publish_date": "2013-01-10"
}'2. Running a Full-Text Search
This is where the magic happens. We’ll search for the word “business” within the summary field. OpenSearch uses the BM25 algorithm to calculate a relevance score for every document it finds.
curl -X GET "http://localhost:9200/books/_search" -H 'Content-Type: application/json' -d'
{
"query": {
"match": {
"summary": "business"
}
}
}'The response includes a _score. If you had 100 books mentioning “business,” the ones where the word is most prominent would appear at the top of the list.
Practical Use Case: Log Analytics
Searching for books is one thing, but troubleshooting a distributed system is another. When you have 50 microservices, SSHing into individual servers to read logs is impossible. By shipping logs into OpenSearch via Fluent Bit, you can search through 100GB of data in milliseconds.
In the Dashboards interface, use the Discover tab to filter logs by service name or error level. You can build a real-time pie chart showing 404 vs. 500 error codes or a line graph tracking request latency. This visibility is the difference between a 5-minute outage and a 5-hour one.
Final Thoughts
Moving from basic SQL queries to a dedicated engine like OpenSearch changes your development workflow. You stop fighting with slow database scans and start building features that users actually enjoy using. Docker makes it trivial to spin up an instance for testing, and because it’s API-compatible with Elasticsearch, you can use almost any existing library or tool in the ecosystem.
Whether you’re building an e-commerce site or a monitoring stack, OpenSearch provides a high-performance, open-source foundation that scales as your data grows.
