OpenSearchHybrid searchLLMAWS Bedrock

Enterprise Search

Fixing semantic search on scientific topics.

Role
Proposed and Built
Where
a Knowledge Discovery Platform
Year
2025

Note: Anonymized. Client and platform related specifics are intentionally left out.

The result

~3×

higher score on the internal search benchmark✱

LLM-powered query expansion using AWS Bedrock and OpenSearch, surfacing scientific documents, for ~3× higher search relevance.

The problem

The company had an internal platform for surfacing assets, tools and utilities built across teams.

It used a hybrid search structure (with a BM25 keyword search + KNN vector search) over OpenSearch hosted on AWS.

The semantic side underperformed, especially on niche topics from scientific literature.

The path not taken

The plan was to replace the embedding model with one that handled scientific terms better, either trained in house or found online.

That meant a long build and evaluation process, the cost of hosting a new model, and no promises of being a complete solution.

I instead proposed a smaller query expansion module. Which would be easier to build and evaluate if it worked.

RISKY

Original plan

Custom embedding model

SHIPPED

Proposed plan

Query expansion module

Effort
✕Train or find a model
✓One small add-on
Running cost
✕Host a model
✓One LLM call per query
Will it work?
✕No guarantee
✓Easy to test first
Index changes
✕Re-embed everything
✓No change required

What got built

We went ahead with the query expansion module. Every query would now get expanded by an LLM on AWS Bedrock before hitting the index.

The module handled both the keyword and vector legs of the search in a single LLM call.

The BM25 query was appended with exact related terms. The KNN vector search received standardized phrases explaining the intent of the user.

The refinement

While integrating this module, the keyword index was found to store each asset's name, descriptions, tags and metadata as one combined field.

These were split into separate fields, and a new query weighted each by relevance. This improved the reliability of keyword matches, especially against typos.

After: name, description, tags and metadata in separate fields, each with its own weight
Before: an asset's name, description, tags and metadata stored together in one text field
drag to compare the index, before and after

Result

Together, the changes scored ~3× higher on an internal search relevance benchmark.

✱ how was this measured?

How it was measured

The platform owners kept a list of search queries and the results they expected.

That list became a benchmark with two scores:

  • Top-1 accuracy: was the right result first?
  • Top-3 score: a right result still counts in second or third place, just for less (1, ½, ⅓).

I learned…

how to evaluate solutions on viability, cost and effort.

Next experience

Intrapartum AI

→