使用Java API獲取Elasticsearch的所有記錄

Question

我想從使用Java API的Elasticsearch獲取所有記錄。但我收到以下錯誤

n [[Wild Thing] [localhost：9300] [indices：data/read/search [phase/dfs]]]; 嵌套：QueryPhaseExecutionException [結果窗口太大，從 +大小必須小於或等於：[10000]，但是[10101]。

我的代碼低於當前存在的有131881條記錄

Client client; 
try { 
    client = TransportClient.builder().build(). 
      addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("localhost"), 9300)); 
    int from = 1; 
    int to = 100; 
    while (from <= 131881) { 
     SearchResponse response = client 
       .prepareSearch("demo_risk_data") 
       .setSearchType(SearchType.DFS_QUERY_THEN_FETCH).setFrom(from) 
       .setQuery(QueryBuilders.boolQuery().mustNot(QueryBuilders.termQuery("user_agent", ""))) 
       .setSize(to).setExplain(true).execute().actionGet(); 
     if (response.getHits().getHits().length > 0) { 
      for (SearchHit searchData : response.getHits().getHits()) { 
       JSONObject value = new JSONObject(searchData.getSource()); 
       System.out.println(value.toString()); 
      } 
     } 
    } 
} 

總數，所以我開始from = 1和to = 100，然後得到100個記錄，直到from <= 131881。有沒有辦法讓我可以檢查獲得100條記錄的記錄，直到Elasticsearch中沒有進一步的記錄爲止。

Answer 1

是的，您可以使用scroll API，這是Java客戶端also supports。

你可以這樣說：

Client client; 
try { 
    client = TransportClient.builder().build(). 
      addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("localhost"), 9300)); 

    QueryBuilder qb = QueryBuilders.boolQuery().mustNot(QueryBuilders.termQuery("user_agent", "")); 
    SearchResponse scrollResp = client.prepareSearch("demo_risk_data") 
     .addSort(SortParseElement.DOC_FIELD_NAME, SortOrder.ASC) 
     .setScroll(new TimeValue(60000)) 
     .setQuery(qb) 
     .setSize(100).execute().actionGet(); 

    //Scroll until no hits are returned 
    while (true) { 
     //Break condition: No hits are returned 
     if (scrollResp.getHits().getHits().length == 0) { 
      break; 
     } 

     // otherwise read results 
     for (SearchHit hit : scrollResp.getHits().getHits()) { 
      JSONObject value = new JSONObject(searchData.getSource()); 
      System.out.println(value.toString()); 
     } 

     // prepare next query 
     scrollResp = client.prepareSearchScroll(scrollResp.getScrollId()).setScroll(new TimeValue(60000)).execute().actionGet(); 
    } 
}