Using image similarity search

Using image similarity search

Image similarity search tool helps you to find similar images in your data set. Unlike other algorithms like image classification, object detection, and semantic/instance segmentation, image similarity search tool does not require neither image labeling nor model training. You can use the image similarity search tool out-of-the-box!

There are two main options to use image similarity Search online or offline. If you want to use the image similarity search online, you can do it via SentiSight.ai web platform or via our REST API server. If you would like to use the model offline, you will have to download the model and set up REST API server yourself. Below we will describe all of those options.

There are two types of Image Similarity Search you can perform: 1vN that finds similar images to a single query image and NvN that finds most similar image pairs in your data set. The query image for 1vN image similarity search can either be uploaded from your computer or selected from your data set on SentiSight.ai platform.

Regardless of the type of image similarity search you are planning to use, first you need to create a project and upload some images that will be used as a data set for your image similarity search queries. See collecting data section for instructions how to do that. Note that for image similarity search image labeling is not necessary.

Note, that every image similarity search performed counts against your monthly prediction limit.

Consider cheking out image similarity search video tutorial.

Using image similarity search tool via SentiSight.ai web platform

To perform the image similarity search, click on the “Pre-trained models" menu item on the top of the screen and select "Image similarity search”. In the menu that pops up, you can select image similarity search type 1vN or NvN. The 1vN type will find you similar images to the query image that you upload. The NvN type will look for similar image pairs in all of your data set. The result number field will adjust how many top matching images/image pairs are displayed and score threshold will determine what is the minimum similarity score for the matching images/image pairs to be displayed.

Note that you can also perform 1vN image similarity search with an existing image in your data set. To do that, just right click on any image and select “Image similarity search” option from a dropdown menu.

Before performing image similarity search you can use “Filter by type” and “Filter by labels” tools on the lower left corner of the main screen. This will narrow down the search space. When filtering images by labels, you can choose between "and" and "or" operators.

Using image similarity search tool via SentiSight.ai REST API

To begin using the image similarity search tool via REST API you will need these details:

  • API token (available under "User profile" menu tab)
  • Project ID (available under "User profile" menu tab)

For 1vN image similarity search you will also need an image file (either existing in your project or stored in your computer).

In all types of image similarity search you can optionally specify the maximum number of results to be displayed and the similarity score threshold. You can also specify one or more image labels to filter the images in your data set prior to image similarity search, hence, reducing the search space. You can also choose between "and" and "or" operators when filtering images by labels. All of these options can be specified as GET parameters when formatting the URL for the request.

For more details see the code samples below.

1vN image similarity search with an image from your computer

Set the "X-Auth-token" header to your API token string and set "Content-Type" header to "application/octet-stream". Set the body to your image file. See the example code below.


TOKEN="your_token"
PROJECT_ID="your_project_id"
IMAGE_FILENAME="your_image_path" # This is the filepath to the image on your PC
FILTER_LABELS="your labels here" # labels should be separated by spaces
AND_OPERATOR="true" # set to "false" to use "or" operator instead for filtering images by labels

LABEL_OP=''
for LABEL in $FILTER_LABELS ; do
  LABEL_OP+="&labels=$LABEL"
done

curl -H "X-Auth-token: $TOKEN" --data-binary @"$IMAGE_FILENAME" \
  -H "Content-Type: application/octet-stream" \
  -X POST "https://platform.sentisight.ai/api/similarity?project=$PROJECT_ID$LABEL_OP&and=$AND_OPERATOR"


import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.nio.file.Files;

public class App {
    public static void main(String[] args) throws IOException {
        String token = "";
        String projectId = "0";
        String imageFilename = "image.jpg";
        String[] labels = {"label1", "label2"};
        int limit = 10;
        double threshold = 0;

        String lstr = "";
        for (String label : labels) {
            lstr += ("&labels=" + label);
        }
        lstr += ("&and=true"); // to use 'or' operator for filtering labels remove this line or change it to '&and=false'
        lstr += ("&limit=" + limit);
        lstr += ("&threshold=" + threshold);

        URL url = new URL("https://platform.sentisight.ai/api/similarity?project=" + projectId + lstr);
        HttpURLConnection connection = (HttpURLConnection) url.openConnection();
        connection.setRequestProperty("Content-Type", "application/octet-stream");
        connection.setRequestProperty("X-Auth-token", token);
        connection.setRequestMethod("POST");
        connection.setDoOutput(true);
        DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
        byte[] bytes = Files.readAllBytes(new File(imageFilename).toPath());
        wr.write(bytes);
        wr.flush();
        wr.close();

        BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
        String output;
        while ((output = in.readLine()) != null) {
            System.out.println(output);
        }
        in.close();
    }
}


<!DOCTYPE html>
<html lang="en" dir="ltr">
    <head>
        <meta charset="utf-8">
        <title>Sample</title>
        <script>
            const baseApiURL = 'https://platform.sentisight.ai/api/';
            var token = '';
            var predictionId;
            var projectId;
            var images = [];

            function similaritySearch() {
                document.getElementById('button').disabled = true;
                token = document.getElementById('tokenfield').value;
                projectId = document.getElementById('project').value;
                var input = document.getElementById('upload');
                const file = input.files[0];
                var fr = new FileReader();
                fr.onload = function() {
                    performSimilaritySearch(fr.result);
                }
                if (file) {
                    fr.readAsArrayBuffer(file);
                }
            }

            function performSimilaritySearch(file) {
                var labelText = document.getElementById('labels').value;
                var labels = [];
                if (labelText.length) {
                    labels = labelText.split(',');
                }
                var lstr = '';
                labels.forEach(function(label) {
                    lstr += ('&labels=' + label);
                });
                if (document.getElementById('limit-on').checked) {
                    lstr += ('&limit=' + document.getElementById('limit').value)
                }
                if (document.getElementById('threshold-on').checked) {
                    lstr += ('&threshold=' + document.getElementById('threshold').value)
                }
                if (document.querySelector('input[name="and"]:checked').value == 'true') {
                    lstr += '&and=true';
                }
                var xmlHttp = new XMLHttpRequest();
                xmlHttp.open('POST',  baseApiURL + 'similarity?project=' + projectId + lstr, true);
                xmlHttp.setRequestHeader('Content-Type', 'application/octet-stream');
                xmlHttp.setRequestHeader('X-Auth-token', token);
                xmlHttp.onreadystatechange = function () {
                    if (this.readyState == 4 && this.status == 200) {
                        document.getElementById('button').disabled = false;
                        document.getElementById('results').innerHTML = xmlHttp.responseText;
                    }
                }
                xmlHttp.send(file);
            }

        </script>
    </head>
    <body>
        Token: <input id="tokenfield" type="text">
        <br>
        Project id: <input id="project" type="number">
        <br>
        Labels: <input id="labels" type="text">
        <br>
        <input type="radio" name="and" value="true"> And
        <input type="radio" name="and" value="false" checked> Or
        <br>
        <input id="limit-on" type="checkbox">Result number: <input id="limit" type="number">
        <br>
        <input id="threshold-on" type="checkbox">Score threshold: <input id="threshold" type="number" step="0.001">
        <br>
        Upload image: <input id="upload" type="file">
        <br>
        <button id="button" type="button" onclick="similaritySearch()">Perform similarity search</button>
        <br>
        <table id=results></table>
    </body>
</html>

import requests

token = "your_token"
project_id = "your_project_id"
image_filename = "your_image_path"
labels = ["your", "labels", "here"]
limit = 10  # limits result count to this number
threshold = 0  # only shows results above this threshold (value in percent)

headers = {"X-Auth-token": token, "Content-Type": "application/octet-stream"}

request_url = "https://platform.sentisight.ai/api/similarity?project={}".format(project_id)
for lbl in labels:
    request_url += "&labels={}".format(lbl)

request_url += "&and=true" # to use 'or' operator for filtering labels remove this line or change it to '&and=false'
request_url += "&limit={}".format(limit)
request_url += "&threshold={}".format(threshold)

with open(image_filename, "rb") as handle:
    r = requests.post(request_url, headers=headers, data=handle)

if r.status_code == 200:
    print(r.text)
else:
    print("Error occured with REST API.")
    print("Status code: {}".format(r.status_code))
    print("Error message: " + r.text)

1vN image similarity search with an existing image in your data set

Set the "X-Auth-token" header to your API token string and set "Content-Type" header to "text/plain". Set the body to your query image name. The query image should be already uploaded to the SentiSight.ai platform. See the example code below.


TOKEN="your_token"
PROJECT_ID="your_project_id"
IMAGE_NAME="your_image_name" # this the name of an image which is already uploaded on platform.sentisight.ai
FILTER_LABELS="your labels here" # labels should be separated by spaces
AND_OPERATOR="true" # set to "false" to use "or" operator instead for filtering images by labels

LABEL_OP=''
for LABEL in $FILTER_LABELS ; do
  LABEL_OP+="&labels=$LABEL"
done

curl -H "X-Auth-token: $TOKEN" --data "$IMAGE_NAME" \
  -H "Content-Type: text/plain" \
  -X POST "https://platform.sentisight.ai/api/similarity?project=$PROJECT_ID$LABEL_OP&and=$AND_OPERATOR"

import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;

public class App {
    public static void main(String[] args) throws IOException {
        String token = "";
        String projectId = "0";
        String imageName = "image.jpg";
        String[] labels = {"label1", "label2"};
        int limit = 10;
        double threshold = 0;

        String lstr = "";
        for (String label : labels) {
            lstr += ("&labels=" + label);
        }
        lstr += ("&and=true"); // to use 'or' operator for filtering labels remove this line or change it to '&and=false'
        lstr += ("&limit=" + limit);
        lstr += ("&threshold=" + threshold);

        URL url = new URL("https://platform.sentisight.ai/api/similarity?project=" + projectId + lstr);
        HttpURLConnection connection = (HttpURLConnection) url.openConnection();
        connection.setRequestProperty("Content-Type", "text/plain");
        connection.setRequestProperty("X-Auth-token", token);
        connection.setRequestMethod("POST");
        connection.setDoOutput(true);
        DataOutputStream wr = new DataOutputStream(connection.getOutputStream());
        wr.writeBytes(imageName);
        wr.flush();
        wr.close();

        BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
        String output;
        while ((output = in.readLine()) != null) {
            System.out.println(output);
        }
        in.close();
    }
}

<!DOCTYPE html>
<html lang="en" dir="ltr">
    <head>
        <meta charset="utf-8">
        <title>Sample</title>
        <script>
            const baseApiURL = 'https://platform.sentisight.ai/api/';
            var token = '';
            var predictionId;
            var projectId;
            var images;

            function similaritySearch(input) {
                document.getElementById('button').disabled = true;
                token = document.getElementById('tokenfield').value;
                projectId = document.getElementById('project').value;
                var input = document.getElementById('image-name').value;
                var labelText = document.getElementById('labels').value;
                var labels = [];
                if (labelText.length) {
                    labels = labelText.split(',');
                }
                var lstr = '';
                labels.forEach(function(label) {
                    lstr += ('&labels=' + label);
                });
                if (document.getElementById('limit-on').checked) {
                    lstr += ('&limit=' + document.getElementById('limit').value)
                }
                if (document.getElementById('threshold-on').checked) {
                    lstr += ('&threshold=' + document.getElementById('threshold').value)
                }
                if (document.querySelector('input[name="and"]:checked').value == 'true') {
                    lstr += '&and=true';
                }
                var xmlHttp = new XMLHttpRequest();
                xmlHttp.open('POST',  baseApiURL + 'similarity?project=' + projectId + lstr, true);
                xmlHttp.setRequestHeader('Content-Type', 'text/plain');
                xmlHttp.setRequestHeader('X-Auth-token', token);
                xmlHttp.onreadystatechange = function () {
                    if (this.readyState == 4 && this.status == 200) {
                        document.getElementById('button').disabled = false;
                        document.getElementById('results').innerHTML = xmlHttp.responseText;
                    }
                }
                xmlHttp.send(input);
            }

        </script>
    </head>
    <body>
        Token: <input id="tokenfield" type="text">
        <br>
        Project id: <input id="project" type="number">
        <br>
        Labels: <input id="labels" type="text">
        <br>
        <input type="radio" name="and" value="true"> And
        <input type="radio" name="and" value="false" checked> Or
        <br>
        <input id="limit-on" type="checkbox">Result number: <input id="limit" type="number">
        <br>
        <input id="threshold-on" type="checkbox">Score threshold: <input id="threshold" type="number" step="0.001">
        <br>
        Image name: <input id="image-name" type="text">
        <br>
        <button id="button" type="button" onclick="similaritySearch()">Perform similarity search</button>
        <br>
        <table id=results></table>
    </body>
</html>

import requests

token = "your_token"
project_id = "your_project_id"
image_name = "your_image_name".encode("utf-8")
labels = ["your", "labels", "here"]
limit = 10  # limits result count to this number
threshold = 0  # only shows results above this threshold  (value in percent)

headers = {"X-Auth-token": token, "Content-Type": "text/plain"}

request_url = "https://platform.sentisight.ai/api/similarity?project={}".format(project_id)
for lbl in labels:
    request_url += "&labels={}".format(lbl)
    
request_url += "&and=true" # to use 'or' operator for filtering labels remove this line or change it to '&and=false'
request_url += "&limit={}".format(limit)
request_url += "&threshold={}".format(threshold)

r = requests.post(request_url, headers=headers, data=image_name)

if r.status_code == 200:
    print(r.text)
else:
    print("Error occured with REST API.")
    print("Status code: {}".format(r.status_code))
    print("Error message: " + r.text)

NvN image similarity search

Set the "X-Auth-token" header to your API token string and set "Content-Type" header to "text/plain".


TOKEN="your_token"
PROJECT_ID="your_project_id"
FILTER_LABELS="your labels here" # labels should be separated by spaces
AND_OPERATOR="true" # set to "false" to use "or" operator instead for filtering images by labels

LABEL_OP=''
for LABEL in $FILTER_LABELS ; do
  LABEL_OP+="&labels=$LABEL"
done

curl -H "X-Auth-token: $TOKEN" \
  -H "Content-Type: text/plain" \
  -X GET "https://platform.sentisight.ai/api/similarity?project=$PROJECT_ID$LABEL_OP&and=$AND_OPERATOR"

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;

public class App {
    public static void main(String[] args) throws IOException {
        String token = "";
        String projectId = "0";
        String[] labels = {"label1", "label2"};
        int limit = 10;
        double threshold = 0;
        
        String lstr = "";
        for (String label : labels) {
            lstr += ("&labels=" + label);
        }
        lstr += ("&and=true"); // to use 'or' operator for filtering labels remove this line or change it to '&and=false'
        lstr += ("&limit=" + limit);
        lstr += ("&threshold=" + threshold);

        URL url = new URL("https://platform.sentisight.ai/api/similarity?project=" + projectId + lstr);
        HttpURLConnection connection = (HttpURLConnection) url.openConnection();
        connection.setRequestProperty("Content-Type", "text/plain");
        connection.setRequestProperty("X-Auth-token", token);
        connection.setRequestMethod("GET");

        BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
        String output;
        while ((output = in.readLine()) != null) {
            System.out.println(output);
        }
        in.close();
    }
}

<!DOCTYPE html>
<html lang="en" dir="ltr">
    <head>
        <meta charset="utf-8">
        <title>Sample</title>
        <script>
            const baseApiURL = 'https://platform.sentisight.ai/api/';
            var token = '';
            var predictionId;
            var projectId;

            function similaritySearch(input) {
                document.getElementById('button').disabled = true;
                token = document.getElementById('tokenfield').value;
                projectId = document.getElementById('project').value;
                var labelText = document.getElementById('labels').value;
                var labels = [];
                if (labelText.length) {
                    labels = labelText.split(',');
                }
                var lstr = '';
                labels.forEach(function(label) {
                    lstr += ('&labels=' + label);
                });
                if (document.getElementById('limit-on').checked) {
                    lstr += ('&limit=' + document.getElementById('limit').value)
                }
                if (document.getElementById('threshold-on').checked) {
                    lstr += ('&threshold=' + document.getElementById('threshold').value)
                }
                if (document.querySelector('input[name="and"]:checked').value == 'true') {
                    lstr += '&and=true';
                }
                var xmlHttp = new XMLHttpRequest();
                xmlHttp.open('GET',  baseApiURL + 'similarity?project=' + projectId + lstr, true);
                xmlHttp.setRequestHeader('Content-Type', 'text/plain');
                xmlHttp.setRequestHeader('X-Auth-token', token);
                xmlHttp.onreadystatechange = function () {
                    if (this.readyState == 4 && this.status == 200) {
                        document.getElementById('button').disabled = false;
                        document.getElementById('result').innerHTML = xmlHttp.responseText;
                    }
                }
                xmlHttp.send();
            }

        </script>
    </head>
    <body>
        Token: <input id="tokenfield" type="text">
        <br>
        Project id: <input id="project" type="number">
        <br>
        Labels: <input id="labels" type="text">
        <br>
        <input type="radio" name="and" value="true"> And
        <input type="radio" name="and" value="false" checked> Or
        <br>
        <input id="limit-on" type="checkbox">Result number: <input id="limit" type="number">
        <br>
        <input id="threshold-on" type="checkbox">Score threshold: <input id="threshold" type="number" step="0.001">
        <br>
        <button id="button" type="button" onclick="similaritySearch()">Perform similarity search</button>
        <br>
        <table id="result"></table>
    </body>
</html>

import requests

token = "your_token"
project_id = "your_project_id"
labels = ["your", "labels", "here"]
limit = 10  # limits result count to this number
threshold = 0.5  # only shows results above this threshold (value in percent)

headers = {"X-Auth-token": token, "Content-Type": "text/plain"}

request_url = "https://platform.sentisight.ai/api/similarity?project={}".format(project_id)
for lbl in labels:
    request_url += "&labels={}".format(lbl)

request_url += "&and=true" # to use 'or' operator for filtering labels remove this line or change it to '&and=false'
request_url += "&limit={}".format(limit)
request_url += "&threshold={}".format(threshold)

r = requests.get(request_url, headers=headers)

if r.status_code == 200:
    print(r.text)
else:
    print("Error occured with REST API.")
    print("Status code: {}".format(r.status_code))
    print("Error message: " + r.text)

Using Image Similarity Search offline - setting up your own REST API server

If you want to set up your own REST API server, you will have to download an offline version of Image Similarity Search model. To do that click "Pre-trained models" -> "Image Similarity Search" -> "Download model".

After the model is downloaded, follow the instructions in Readme.md to set up your local REST API server. You can make the client requests from the same PC on which you set up the server, so the model would be run completely offline. On the other hand, after you set up REST API server, you can also make client requests to this server from many different devices (including mobile) on your network. Note that the REST API server must be run on a Linux system, but the client devices can run on any operating system.

The offline version of the model can be run as a free trial for 30 days. After this period, if you like the offline version, you will have to buy a license from us. Note that we have three options for the speed of the offline model: slow, medium and fast. The license price depends on the selected speed. The free trial always runs on the slow speed mode.