Search Request

In this workflow, it is possible that the document written to the vBucket is not yet indexed. So, when FTS starts searching that document, the most up-to-date document versions are not retrieved, and only the indexed versions are queried.

Therefore, the Couchbase server provides a consistency mechanism to overcome this issue and ensures that the FTS index can search the most up-to-date document written to the vBucket.

The consistency level is a parameter that either takes an empty string indicating the unbounded (not_bounded) consistency or at_plus indicating the bounded consistency.

The consistency vectors supporting the consistency mechanism in Couchbase contain the mapping of the vBucket and sequence number of the data stored in the vBucket.

For more information about consistency mechanism, see Consistency

It is the amount of time (in milliseconds) the search service will allow for a query to execute at an index partition level.

If the query execution surpasses this timeout value, the query is canceled. However, at this point if some of the index partitions have responded, you might see partial results, otherwise no results at all.

Consistency result is the attribute that you can use to set the query result option, such as complete.

The complete option allows you to set the query result as "complete" which indicates that if any of the index partitions are unavailable due to the node not being reachable, the query will display an error in response instead of partial results.

Consistency vectors provide "read your own writes" functionality where the read operation waits for a specific time until the write operation is finished.

When users know that their queries are complex which require more time in completing the write operations, they can set the timeout value higher than the default timeout of 10 seconds so that consistency can be obtained in the search operations.

However, if this consistency is not required, the users can optimize their search operations by using the default timeout of 10 seconds.

This pagination settings can be used to obtain a subset of results and works deterministically when combined with a certain sort order.

Using size/limit and offset/from would fetch at least size + from ordered results from a partition and then return the size number of results starting at offset from.

Deep pagination can therefore get pretty expensive when using size + from on a sharded index due to each shard having to possibly return large resultsets (at least size + from) over the network for merging at the coordinating node before returning the size number of results starting at offset from.

The default sort order is based on score (relevance) where the results are ordered from the highest to the lowest score.

Here are some examples using search_after/search_before over sort key "_id" (an internal field that carries the document ID).

The pagination of search results can be done using the from and size parameters in the search request. But as the search gets into deeper pages, it starts consuming more resources.

To safeguard against any arbitrary higher memory requirements, FTS provides a configurable limit bleveMaxResultWindow (10000 default) on the maximum allowable page offsets. However, bumping this limit to higher levels is not a scalable solution.

To circumvent this problem, the concept of key set pagination in FTS, is introduced.

Instead of providing from as a number of search results to skip, the user will provide the sort value of a previously seen search result (usually, the last result shown on the current page). The idea is that to show the next page of the results, we just want the top N results of that sort after the last result from the previous page.

This solution requires a few preconditions be met:

A common solution to this is to always include the document ID as the final sort criteria.

For example, if you want to sort by [“name”, “-age”], instead of sort by [“name”, “-age”, "_id"].

With search_after/search_before paginations, the heap memory requirement of deeper page searches is made proportional to the requested page size alone. So it reduces the heap memory requirement of deeper page searches significantly down from the offset+from values. .

FTS sorting is sorted by descending order of relevance. It can, however, be customized to sort by different fields, depending on the application.

On query-completion, sorting allows specified members of the result-set to be displayed prior to others: this facilitates a review of the most significant data.

Within a JSON query object, the required sort-type is specified by using the sort field.

This takes an array of either strings, objects, or numeric as its value.

You can specify the value of the sort field as an array of strings. These can be of three types:

Fine-grained control over sort-procedure can be achieved by specifying objects as array-values in the sort field.

Each object can have the following fields:

To fetch more accurate sort results, we strongly recommend specifying the type of the sort fields in the sort section of the search request.

You can specify the value of the sort field as a numeric type. You can use the type field in the object that you specify with the sort.

With type field, you can specify the type of the search order to numeric, string, or DateTime.

When you sort results on a field that is not indexed, or when a particular document is missing a value for that field, you will see the following series of Unicode non-printable characters appear in the sort field:

\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd

The same characters may render differently when using a graphic tool or command line tools like jq.

Check your index definition to confirm that you are indexing all the fields you intend to sort by. You can control the sort behavior for missing attributes using the missing field.

The following sample query response shows the score field for each document retrieved for the query request:

tf-idf, a short form of term frequency-inverse document frequency, is a numerical statistical value that is used to reflect how important a word is to a document in collection or scope.

tf-idf is used as a weighting factor in a search for information retrieval and text mining. The tf–idf value increases proportionally to the number of times a word appears in the document, and it is offset by the number of documents in the collection or scope that contains the word.

Search engines often use the variations of tf-idf weighting scheme as a tool in scoring and ranking a document’s relevance for a given query. The tf-idf scoring for a document relevancy is done on the basis of per-partition index, which means that documents across different partitions may have different scores.

When bleve scores a document, it sums a set of sub scores to reach the final score. The scores across different searches are not directly comparable as the scores are directly dependent on the search criteria. So, changing the search criteria, like terms, boost factor etc. can vary the score.

The more conjuncts/disjuncts/sub clauses in a query can influence the scoring. Also, the score of a particular search result is not absolute, which means you can only use the score as a comparison to the highest score from the same search result.

FTS does not provide any predefined range for valid scores.

In Couchbase application, you get an option to explore the score computations during any search in FTS.

On the Search page, you can search for a term in any index. The search result displays the search records along with the option Explain Scoring to view the score deriving details for search hits and which are determined by using the tf-idf algorithm.

You can disable the scoring by setting score to none in the search request. This is recommended in a situation where scoring (document relevancy) is not needed by the application.

For a select term, FTS calculates the relevancy score. So, the documents having a higher relevancy score automatically appear at the top in the result.

It is often observed that users are using Full-Text Search for the exact match queries with a bit of fuzziness or other search-specific capabilities like geo.

Text relevancy score does not matter when the user is looking for exact or more targeted searches with many predicates or when the dataset size is small.

In such a case, FTS unnecessarily uses more resources in calculating the relevancy score. Users can, however, optimize the query performance by skipping the scoring. Users may skip the scoring by passing a “score”: “none” option in the search request.

As per the following example, when you search the content in the index, the matched content in the address field is highlighted in the search response.

As per the following example, when you search the content in the index, the matched content in the description field is highlighted in the search response.

A term facet counts how many matching documents have a particular term for a specific field.

A numeric range facet works by the users defining their own buckets (numeric ranges).

The facet then counts how many of the matching documents fall into a particular bucket for a particular field.

The Date Range facet is same as numeric facet, but on dates instead of numbers.

Full text search and Bleve expect dates to be in the format specified by RFC-3339, which is a specific profile of ISO-8601 that is more restrictive.