Remove 10000 limit when using elasticsearch

  • wpsolr
    Keymaster
    2 years, 10 months ago #18819

    @maxwatson Can you show me exactly the problem in a screenshot?

    maxwatson
    Participant
    2 years, 10 months ago #18821

    Sure, I’ve tried to cover all the scenarios in this image, please have a look and let me know if there’s anything else you need. Maybe I’ve missed a setting somewhere.

    https://ibb.co/bBv9nsH

    wpsolr
    Keymaster
    2 years, 10 months ago #18822

    Clients with the same WPSOLR as yours can display hundreds of thousands of results. No need of Elasticsearch cursors. Cursors are only useful for paginating above 10K results (displaying search page 1000, if a page contains 10 results). But you are showing page 1.

    wpsolr
    Keymaster
    2 years, 10 months ago #18823

    It could be due to a breaking change for Elasticsearch 7, which returns 10,000 total documents maximum, unless track-total-hits is set on each query:
    https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#track-total-hits-10000-default

    maxwatson
    Participant
    2 years, 10 months ago #18827

    Yes, it’s true, if I set track_total_hits=true, i get all the results
    “hits” : {
    “total” : {
    “value” : 176275,
    “relation” : “eq”
    },
    But how to add this parameter to each query?

    It isn’t possible to add it via configuration, i need to set it on every request.

    maxwatson
    Participant
    2 years, 10 months ago #18828

    I see for other plugin you can add it by a function, how can I modify it for WPSOLR?

    add_filter( ‘ep_formatted_args’, function ( $formatted_args ) {
    $formatted_args[‘track_total_hits’] = true;
    return $formatted_args;
    } );

    wpsolr
    Keymaster
    2 years, 10 months ago #18844

    Good idea. I moved the topic in “New feature requests” forum.

    wpsolr
    Keymaster
    2 years, 6 months ago #21347

    The detailed tutorial to fix the 10,000 results in Elasticsearch is on Fix Elasticsearch error “Result window is too large”

    mneimne
    Participant
    9 months, 2 weeks ago #29617

    Hello
    I followed your guide on how to make Elasticsearch return more than 10,000 results, however it doesn’t seem to be working, always 10K results are returned.
    Knowing that the settings API call on that index indicates that this value was set to more than 10K.
    Elasticsearch version 7.6.2

    wpsolr
    Keymaster
    9 months, 2 weeks ago #29619

    Indeed, Elasticsearch settings as described in our documentation do not work anymore with the recent versions of Elasticsearch / OpenSearch.

    Here is a fix, released in next version of WPSOLR) with a new option “Show all results” in screen 2.1: https://trello.com/c/RYsQ9hr8/189-add-option-to-show-all-results-with-elasticsearch: it sets the parameter ‘track_total_hits’ to ‘true’ for all queries.

    (It has been tested on our Kinsta demo with 100,000 products: https://demo-woocommerce-flatsome-kinsta-100k-products.wpsolr.com/shop/ )

    wpsolr
    Keymaster
    5 months, 2 weeks ago #30628

    The fix is now part of WPSOLR 22.3: you can override the Elasticsearch 10,000 default limit from the view’s configuration.

    Image wpsolr-admin-remove-10000-limit-1024x506.png of

Viewing 11 posts - 16 through 26 (of 26 total)

You must be logged in to reply to this topic.