Why sorting in elasticSearch does not sort data properly?

Question

First Question:- I have data of size around 45000. I want to sort that data on chrom and pos key. I have written the query to sort data shown below.

   //The below script sort the chromosomes 
   SortBuilder builder=new ScriptSortBuilder("s = doc['chrom'].value; s=s.substring(3); s.indexOf('X')!=-1?23:s.indexOf('Y')!=-1?24:s.indexOf('MT')!=-1?25:s.indexOf('M')!=-1?25:s;" +
                    "n = org.elasticsearch.common.primitives.Ints.tryParse(s); if (n != null) { String.format(\"%010d\",n)} else { s }", String.class.getSimpleName().toLowerCase());
   SearchRequestBuilder setQuery = this.getClient().prepareSearch(this.getIndex()).setTypes(this.getType())
                    .addSort(builder)
                    .addSort(Keys.POS.toLowerCase(),SortOrder.ASC).
                    setQuery(QueryBuilders.matchQuery(Keys.SAMPLE_ID_DB_KEY, entityID.toLowerCase())).setSize(100).setSearchType(SearchType.QUERY_AND_FETCH).setScroll(new TimeValue(60000000));

However, after firing the query I received multiple bunch of data. Where bunch is sorted but irrespective of data in other bunch.(i.e. If there is entry of 1:11111 present in 1st bunch then there can be entry in second bunch having value less than 1:11111 ).

Am I missing somthing?

Second question:- When I do not specify the size in query it does not returns me all 45000 entries. Why is it so?

Edit
Data in JSON format

{
  "chrom": "chr1",
  "pos": 762273,
  "isIndel": false,
  "interpretation": "",
  "sampleID": "xyz",
  "isSignedOff": false,
  "ownerID": null,
  "entityType": 0
}

Could you add some test data and mapping to your question?

imotov
– imotov

2013-10-11 10:32:50 +00:00
Commented Oct 11, 2013 at 10:32 — imotov
– imotov, Commented Oct 11, 2013 at 10:32
@imotov: I have added test data

Sachin
– Sachin

2013-10-11 11:27:56 +00:00
Commented Oct 11, 2013 at 11:27 — Sachin
– Sachin, Commented Oct 11, 2013 at 11:27

imotov · Accepted Answer · 2013-10-11 12:18:59Z

1

Switch to SearchType.QUERY_THEN_FETCH instead of SearchType.QUERY_AND_FETCH.

answered Oct 11, 2013 at 12:18

imotov

30.2k3 gold badges93 silver badges87 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Sachin Over a year ago

Could you please put some light on the difference between these two?

imotov Over a year ago

@SachinPasalkar see elasticsearch.org/guide/en/elasticsearch/reference/current/…

Collectives™ on Stack Overflow

Why sorting in elasticSearch does not sort data properly?

1 Answer 1

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Related