Search API

The Search API provides personalized, typo-correcting, semantic search for your site. You send this API the search queries users entered, and the API returns the relevant search results tailored to your users’ interests.

Personalized search

Personalized search is a key factor in driving search conversion on many major sites. It is particularly powerful for short search queries (≤ 3 keywords), which account for up to 80% of search traffic in the U.S., but are usually the hardest to get right with traditional search engines. This is because shorter search queries tend to match a larger number of results, but there is not enough information in the query strings alone to determine which results the users are actually looking for.

For example, when users search for jeans on Levi’s.com, it is impossible to know which jeans the user is looking for, among thousands of options. Even if the user adds: jeans for men, it is still unclear to a traditional search engine what style, material, or size the user wants.

In the contrary, with Miso’s personalized search, we not only analyze the search query itself, but also take into account the context in which the searches are made, including who are the users, where are they from, what are their past interactions on the site, what other searches the user made, etc. These signals together allow Miso to generate more than 15% to 20% higher search conversion rate than the traditional non-personalized search engines.

Balancing relevancy and personalization

Although personalization is a powerful technique, over-using it can be harmful to the user experiences. In the context of search optimization, the relevancy of the search results are still the most important criteria, and we don’t want personalization to overwhelm the search relevancy. For example, when users search for a very specific term, or directly search for the product names, Miso’s algorithm will respond with the most relevant search results first, and then only apply personalization to rerank more ambiguous search results.

Basic usage

For every search query, you let Miso know the user’s user_id and the search keywords in the API request body, for example:

POST /v1/search/search
{
  "q": "jeans",
  "user_id": "user-123"
}

For site visitors who do not sign in, you can let Miso know the anonymous_id of this visitor:

POST /v1/search/search
{
  "q": "jeans",
  "anonymous_id": "visitor-123"
}

Search response

With the query above, Miso responds with the search results like the following:

{
   "message":"success",
   "data":{
      "took":50,
      "total":30,
      "start":0,
      "miso_id":"f34b90de-086b-11eb-b498-1ee8abb1818b",
      "products":[
         {
            "product_id":"505-regular-fit-mens-jeans",
            "title":"The 505 Regular Fit Men's Jeans",
            "url":"https://levi.com/jeans/505-regular-fit-mens-jeans/",
            "size":"29",
            "material":"Cotton",
            "color":"Rinse - Dark Wash",
            "_search_score": 78.12,
            "_personalization_score": 0.98
         }
      ],
      "spellcheck":{
         "spelling_errors":false
      }

   }
}

• took: the amount of time (in milliseconds) Miso took to answer the query
• total: the total number of matched products. You can paginate through all the products by using the combination of start and rows parameters (see Request Body Schema below)
• miso_id: a UUID of the search request. You should include miso_id in the Interaction records for every interactions that result from this search request, e.g. user click-through a product in the search results. Miso use miso_id to track the search performance and fine-tune the algorithm accordingly.
• products: an array of Product records that match the search query, ranked in the order of relevancy and probability that the user will be interested in this product. By default, only the product_id of the Product is returned. You can ask Miso to return additional fields by using the fl parameter (see Request Body Schema below)
• products[ ]._search_score: the search relevancy score of the products based on keyword matching and Miso’s semantic matching. This score is similar to traditional Lucene search score.
• products[ ]._personalization_score: the score assigned by Miso’s personalization algorithm based on users’ profile and their interactions on the site. This score quantifies the probability of whether users will be interested in this product or not.
• spellcheck: an dictionary contains spell checking information.

Spellcheck and auto-correction

According to a Microsoft Research study, roughly 10-15% of the queries sent to search engines contain errors. A misspelled search keyword often results in poor search quality, and users have been accustomed to Google’s automatic spelling correction functionality and expect the same on your site.

However, correcting spelling and typos at scale is a non-trivial machine learning problem. Miso’s spellcheck is based on a sequence-to-sequence deep learning model, trained and updated regularly on a corpus of billion tokens. It detects hard-to-spot errors, auto-correct keywords according to its context, and recognize terms that are newer or lesser known.

Spellcheck is always on for every search request so you don’t need to turn it on. What you need to decide is whether to turn on auto spelling correction. For example, the following search request turns on the auto-spelling-correction, and Miso will automatically replace any misspelled queries with their correct spelling:

POST /v1/search/search
{
   "q":"whte denem jeans",
   "user_id":"user-123",
   "spellcheck":{
      "enable_auto_spelling_correction":true
   }
}

The API will respond:

{
   "message":"success",
   "data":{
      "took":50,
      "total":30,
      "start":0,
      "miso_id":"f34b90de-086b-11eb-b498-1ee8abb1818b",
      "spellcheck":{
         "spelling_errors":true,
         "auto_spelling_correction":true,
         "original_query":"whte denem jeans",
         "original_query_with_markups":"<mark>whte</mark> <mark>denem</mark> jeans",
         "corrected_query":"white denim jeans",
         "corrected_query_with_markups":"<mark>white</mark> <mark>denim</mark> jeans"
      },
      "products":[
         ......
      ]
   }
}

The spellcheck object contains the following fields:

• spelling_errors indicates whether there is a spelling error in the query
• auto_spelling_correction indicates whether the search query has been replaced with the corrected_query
• original_query the original search query
• original_query_with_markups the original search query with the misspelled words highlighted by <mark> html tags
• corrected_query the search query with misspelling and typos corrected
• corrected_query_with_markups the search query with misspelling and typos corrected, and the corrected parts are highlighted by <mark> html tags

You can opt-out the auto-spelling-correction by setting enable_auto_spelling_correction=false. For example:

POST /v1/search/search
{
   "q":"whte denem jeans",
   "user_id":"user-123",
   "spellcheck":{
      "enable_auto_spelling_correction":false
   }
}

In this case, Miso will still run spellcheck against the query. However, users’ queries will be used as it is, and auto_spelling_correction field will be false.

Boosting and Diversification

While Miso’s personalized search can drive conversion by showing search results that are tailored to users’ interests, ultimately, it is important to make sure that the search results meet your business goals. To that end, Miso provides a great set of tools that enable you to fine-tune the search ranking and make it aligned with your goals.

One great example is boosting. Boosting allows you to define a query that can be used to boost a subset of products to the top of the ranking, or to specific boost positions. You can use boosting to run different kinds of promotion campaigns, or to promote certain set of products for individual users that you know they will be interested in.

For example, consider a scenario where you need to promote the sales of Nike’s products. Then, you might want to use the query below, that will promote the sneakers whose brand are Nike to the top of the search result:

POST /v1/search/search
{
   "q":"sneaker",
   "user_id":"user-123",
   "boost_fq": "brand:\"Nike\""
}

For a slightly more complex example, the query below will promote the Nike products which have also been tagged as ON SALE:

POST /v1/search/search
{
   "q":"sneaker",
   "user_id":"user-123",
   "boost_fq": "brand:\"Nike\" AND tags:\"ON SALE\""
}

You can have as complex boosting logic as you want in the boosting query, but it is worth mentioning that Miso will only boost products that are relevant and have high likelihood to convert. In other words, Miso will not boost low performance products even if they match the boosting query.

Depending on your boosting rules, in certain cases, you would like to prevent search results from becoming too “plain” due to boosting. For example, you don’t want the first page of the search result to contain only Nike products.

With Miso, you have two tools to avoid so. First, you can specify boost_positions to place boosted products at specific positions in the ranking. For example, the query below will place boosted products only at the first, fourth, seventh places in the ranking (positions are 0-based), and place the remaining products in their original ranking, skipping these three positions.

POST /v1/search/search
{
   "q":"sneaker",
   "user_id":"user-123",
   "boost_fq": "brand:\"Nike\" AND tags:\"ON SALE\"",
   "boost_positions": [0, 3, 6]
}

The second tool is diversification. Miso’s diversification algorithm will maintain a desired minimum distance between any two products that have the same attributes. For example, the following query will make sure products made by the same brand are at least two slots apart from each other in the search results.

POST /v1/search/search
{
   "q":"sneaker",
   "user_id":"user-123",
   "boost_fq": "brand:\"Nike\" AND tags:\"ON SALE\"",
   "diversification": {
       "brand": {"minimum_distance": 2}
    }
}

It is also very often to use both “boost_positions” and “diversification” at the same time to make sure that (1) the search results are not overwhelmed by the boosted products, and (2) there is a good mix of products from different brands showing side-by-side to increase product discovery rate.

Result ordering

You can override Miso’s default ranking order by specifing a list of fields for Miso to rank the search results. These fields can be any numeric or boolean fields in your Product catalog, or one of the following special fields:

• _personalization_score: the score that estimates the probability that a user will interact with a product determined by Miso’s personalization algorithm. The range of this score is between [0, 1]. The scores are non-uniformly distributed. The Products that are relevant to users’ interests will have scores much closer to 1, than products that are not.
• _search_score: the score that rates the degree of “match” between search keywords and a product’s catalog with a focus on Product’s titles. This score is mostly based on a variant of BM25, but additionally consider the term proximity, typos, term semantic similarity. Its value is always larger than 0, but its range is unbounded.
• _boosting_score: a binary score indicates whether a Product is boosted by your boosting query.
• _geo_distance: distance between any point on map, geo must be specified when sorting with this field.

For example, the following query returns all the Products (because q=*), ranked by the _personalization_score first, and then by the values in the custom_attributes.promote_score field in the Product catalog, then the distance between the product and New York city.

{
  "q": "*",
  "order_by": [
        {
            "field": "_personalization_score",
            "tie_breaker": {
                "type": "relative_difference",
                "threshold": "0.05"
            },
            "order": "desc"
        },
        {
            "field": "custom_attributes.promote_score",
            "order": "desc"
        },
        {
            "field": "_geo_distance",
            "geo": {
                "lat": 40.711967,
                "lon": -74.006076,
            }
            "order": "asc"
        }
  ]
}

Mathematical Functions

Miso supports mathematical functions that transform and combine different sorting criteria into one. For example, a powerful strategy to improve gross merchandise volume (GMV), but maintain user experience is to sort the products based on the multiplication of personalization scores and product prices. You can achieve this with the following order_by query:

{
  "q": "*",
  "order_by": [
        {
            "field": "_personalization_score * pow(sale_price, 0.5)",
            "order": "desc"
        }

  ]
}

Function pow(sale_price, 0.5) takes the square root of the sale price and avoids very expensive products from overwhelming the ranking.

Miso supports all the common mathematical operators including +, -, *, /, %, ^, **, and more advanced functions including:

• Power functions: pow(X, y), sqrt(X)
• Exponents and logarithms: exp(X), log(X), log2(X), log10(X)
• Element-wise maximum / minimum: maximum(X, y), minimum(X, y)
• Absolute function: abs(X)
• Rounding functions: round(X), floor(X), ceil(X)
• Trigonometric functions: sin(X), cos(X), tan(X), asin(X), acos(X), atan(X)

Soft Tie-Breaker

For scores that have granular resolutions, for example _personalization_score,_search_scores, or Products’ sale_price, we usually don’t want to rank Products by their raw values. After all, a 0.001 difference in _personalization_score or $0.01 difference in sale price typically will not make a difference in users’ preferences. In such cases, soft tie-breakers should be used to smooth out these minor differences in scores.

For example, in the query above, we apply a soft tie-breaker to _personalization_score based on score values’ relative difference. Specifically, we first sort the score’s raw values in the descending order, then for two consecutive values, if their relative difference is no more than a pre-defined threshold (in this case 0.05 or 5%), they are considered as a tie, and the next field (i.e. custom_attributes.promote_score) will be used to determine their ranking.

It is also common to utilize tie-breakers to combine the effect of two types of scores. For example, in the following query, we set threshold=0.2 or 20% for _personalization_score, then only the Products that users are 20% more likely to interact with will be ranked higher, the remaining Products will be ranked by their sale prices. In this way, we combine the effect of personalization score and sale prices, where the Products are roughly ranked by personalization, but favor the pricier products when they have comparable personalization scores.

{
  "q": "*",
  "order_by": [
        {
            "field": "_personalization_score",
            "tie_breaker": {
                "type": "relative_difference",
                "threshold": "0.20"
            },
            "order": "desc"
        },
        {
            "field": "sale_price",
            "order": "desc"
        }

  ]
}

Also note that, when search keywords are present, it is recommended to always include _search_score as the first field (plus a tie-breaker) to maintain the relevance of the search results. A tie-breaker is usually required as well to let the subsequent score have effect to the ranking.

{
  "q": "toy story",
  "order_by": [
      {
            "field": "_search_score",
            "tie_breaker": {
                "type": "relative_difference",
                "threshold": "0.20"
            },
            "order": "desc"
        },
        {
            "field": "_personalization_score",
            "tie_breaker": {
                "type": "relative_difference",
                "threshold": "0.20"
            },
            "order": "desc"
        },
        {
            "field": "sale_price",
            "order": "desc"
        }

  ]
}