Get started with vector search in Elasticsearch Serverless
Serverless
Elasticsearch enables you to generate mathematical representations of your content called embeddings or vectors. There are two types of representation (dense and sparse), which are suited to different types of queries and use cases (for example, finding similar images and content or storing expanded terms and weights). In this introduction to vector search, you'll store and search for dense vectors. The primary use case for dense vectors is to find pieces of content with similar meanings by using mathematical functions, in this case an approximate k-nearest neighbour(kNN) search.
To learn more about which type of vector is appropriate for your use case, check out Vector search. For an overview of the differences between semantic search and vector search, go to AI-powered search.
To try out vector search, create an Elasticsearch Serverless project that is optimized for vectors.
There are some simple data sets that you can use for learning purposes. For example, if you follow the guided index flow, you can choose the vector search option. Follow the instructions to install an Elasticsearch client and define field mappings. Alternatively, try out the API requests in the Console:
PUT /vector-index/_mapping {
"properties": {
"vector": {
"type": "dense_vector",
"dims": 3
},
"text": {
"type": "text"
}
}
}
This example defines two fields: a three-dimensional dense_vector field and a text field.
Next, use the Elasticsearch bulk API to ingest an array of documents into the index. For example:
POST /_bulk?pretty { "index": { "_index": "vector-index" } }
{"vector":[5.936,3.083,5.087],"text":"Yellowstone National Park is one of the largest national parks in the United States. It ranges from the Wyoming to Montana and Idaho, and contains an area of 2,219,791 acress across three different states. Its most famous for hosting the geyser Old Faithful and is centered on the Yellowstone Caldera, the largest super volcano on the American continent. Yellowstone is host to hundreds of species of animal, many of which are endangered or threatened. Most notably, it contains free-ranging herds of bison and elk, alongside bears, cougars and wolves. The national park receives over 4.5 million visitors annually and is a UNESCO World Heritage Site."}
{ "index": { "_index": "vector-index" } }
{"vector":[4.938,7.78,3.88],"text":"Yosemite National Park is a United States National Park, covering over 750,000 acres of land in California. A UNESCO World Heritage Site, the park is best known for its granite cliffs, waterfalls and giant sequoia trees. Yosemite hosts over four million visitors in most years, with a peak of five million visitors in 2016. The park is home to a diverse range of wildlife, including mule deer, black bears, and the endangered Sierra Nevada bighorn sheep. The park has 1,200 square miles of wilderness, and is a popular destination for rock climbers, with over 3,000 feet of vertical granite to climb. Its most famous and cliff is the El Capitan, a 3,000 feet monolith along its tallest face."}
{ "index": { "_index": "vector-index" } }
{"vector":[9.863,8.919,2.368],"text":"Rocky Mountain National Park is one of the most popular national parks in the United States. It receives over 4.5 million visitors annually, and is known for its mountainous terrain, including Longs Peak, which is the highest peak in the park. The park is home to a variety of wildlife, including elk, mule deer, moose, and bighorn sheep. The park is also home to a variety of ecosystems, including montane, subalpine, and alpine tundra. The park is a popular destination for hiking, camping, and wildlife viewing, and is a UNESCO World Heritage Site."}
In this simple example, the vectors are provided in each document. In a real-world scenario, you could generate the vectors as you added data by using an ingest pipeline with an inference processor.
Now try a query to get the documents that are closest to a vector.
For example, use a knn query with a vector [2,6,0]:
GET vector-search/_search {
"knn": {
"field": "vector",
"k": 10,
"num_candidates": 100,
"query_vector": [2,6,0]
}
}
The search results are sorted by relevance score, which measures how well each document matches the query. For example:
{
"took": 9,
"timed_out": false,
"_shards": {
"total": 3,
"successful": 3,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 3,
"relation": "eq"
},
"max_score": 0.95106244,
"hits": [
{
"_index": "search-05ro",
"_id": "QZqVqZcBabT17nUqiiKf",
"_score": 0.95106244,
"_source": {
"vector": [
4.938,
7.78,
3.88
],
"text": "Yosemite National Park is a United States National Park, covering over 750,000 acres of land in California. A UNESCO World Heritage Site, the park is best known for its granite cliffs, waterfalls and giant sequoia trees. Yosemite hosts over four million visitors in most years, with a peak of five million visitors in 2016. The park is home to a diverse range of wildlife, including mule deer, black bears, and the endangered Sierra Nevada bighorn sheep. The park has 1,200 square miles of wilderness, and is a popular destination for rock climbers, with over 3,000 feet of vertical granite to climb. Its most famous and cliff is the El Capitan, a 3,000 feet monolith along its tallest face."
}
},
...
Thanks for taking the time to try out vector search in Elasticsearch Serverless. For another dense vector example, check out Bring your own dense vectors. For an example of using pipelines to generate text embeddings, check out Tutorial: Dense and sparse workflows using ingest pipelines. To learn about more options, such as semantic and keyword search, go to Search approaches.