Remove 10000 limit when using elasticsearch

  • maxwatson
    Participant
    # 2 years, 6 months ago

    Hello, any ideas how can I remove this limit when using WPSOLR with Elasticsearch? My store has 150K products and when I build the index, I can only see the most recent 10K indexed products.
    Thanks!

    wpsolr
    Keymaster
    maxwatson
    Participant
    # 2 years, 6 months ago

    I see they recommend here using ‘scroll’, but how and where can I apply this to WPSOLR?

    wpsolr
    Keymaster
    # 2 years, 6 months ago

    I added Elasticsearch scroll in the latest beta release, to be able to generate huge Yoast sitemaps (million+).

    What would be your need?

    maxwatson
    Participant
    # 2 years, 6 months ago

    Currently I have WPSOLR PRO 21.8, how can I get the beta release?
    The problem is that the category products are limited to 10K, which is a problem for bigger categories.
    Also I think facets filtering is affected by this also, because some terms don’t appear in filters.

    wpsolr
    Keymaster
    # 2 years, 6 months ago

    The beta is for sitemaps only, because of a limitation of Google’s maximum number of files in a sitemaps (less files means more links in a file, sometimes more than 10K).
    But for category products, you should have a pagination to limit the number of results fetched.

    For facets, they should not be affected by the number of results. Clients’ sites with 500K products or a million listings can display all facets. Perhaps you can increase the option “No. of values to be displayed by filters” in screen 2.1?

    maxwatson
    Participant
    # 2 years, 6 months ago

    Yes, for categories I already have pagination activated, 12 products per page.

    maxwatson
    Participant
    # 2 years, 6 months ago

    Also the weird thing is that I’ve set No. of values to be displayed by filters: 0 for unlimited values and I’m still getting 10 results in sidebar filters.
    I’ve replaced front-end archives for: Search Home/Blog Categories Tags Post types

    maxwatson
    Participant
    # 2 years, 6 months ago

    In error.log I’m getting this error:

    [error] 22638#0: *2675819 FastCGI sent in stderr: “PHP message: WPSOLR message: {“error”:{“root_cause”:[{“type”:”illegal_argument_exception”,”reason”:”Result window is too large, from + size must be less than or equal to: [10000] but was [17528]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.”}],”type”:”search_phase_execution_exception”,”reason”:”all shards failed”,”phase”:”query”,”grouped”:true,”failed_shards”:[{“shard”:0,”index”:”XXX”,”node”:”fwq0djM1XXXizUMgOkT3A”,”reason”:{“type”:”illegal_argument_exception”,”reason”:”Result window is too large, from + size must be less than or equal to: [10000] but was [17528]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.”}}],”caused_by”:{“type”:”illegal_argument_exception”,”reason”:”Result window is too large, from + size must be less than or equal to: [10000] but was [17528]. See the scro…PHP message: WPSOLR trace:

    maxwatson
    Participant
    # 2 years, 6 months ago

    For example if I access directly the search url like this:

    index_name/_search?scroll=1m

    I get
    “hits” : {
    “total” : {
    “value” : 195382,
    “relation” : “eq”
    },

    Without scroll=1m parameter I get

    “hits” : {
    “total” : {
    “value” : 10000,
    “relation” : “gte”
    },

    So how can I add this scroll parameter to the plugin?

    wpsolr
    Keymaster
    # 2 years, 6 months ago

    I do not understand why you get so many results on one single query. The pagination size in screen 2.2 should limit the results to 20 or something around.

    maxwatson
    Participant
    # 2 years, 6 months ago

    This limit appears only on store categories (or other pages like brands or product tags) with over 10K products.
    Pagination from tab 2.1 works perfect, if I set 12 or 48 products per page, I get that number on the front also.
    For search results I didn’t notice this limitation because until now I didn’t have such a big number of results.

    Also in tab 2.1 Replace front-end archives I’ve checked only Search, but it applies the query to product categories also, is this normal?
    I see Warning: permalinks must be activated, but I don’t know where to activate them.

    I’ve tried modifying index.max_result_window in ES to 500000, but nothing changes.

    wpsolr
    Keymaster
    # 2 years, 6 months ago

    “Also in tab 2.1 Replace front-end archives I’ve checked only Search, but it applies the query to product categories also, is this normal?”
    Because “Replace categories search” is selected on the WPSOLR WooCommerce extension?

    maxwatson
    Participant
    # 2 years, 6 months ago

    Yes, this is true, I’ve unchecked Replace WooCommerce category and shop search with WPSOLR’s but now I have some new problems.
    Now the sidebar WPSOLR facets are all gone. This is not very good, specially because I’ve had there the price filter from WPSOLR and it was really cool.
    Also the sort by items don’t work as expected. I have sort by _sale_price_ascending, _sale_price_descending, newest and oldest, when I try to sort by something, nothing happens.

    As soon as I check Replace WooCommerce category and shop search with WPSOLR’s, the sidebar facets appear in product categories and sorting also works like a charm. But also the 10K limit is back in place and I can only get 10K products in those categories.

    I think the best solution is to leave it checked, but what can be done about this awful 10K limit?

    dnikola
    Participant
    # 2 years, 6 months ago

    I am interested in same question.
    I am trying to replace wp admin post edit page on site with 600 000 + posts, but i can see only 10000 when I open All posts page https://prntscr.com/ro2kaz

Viewing 15 posts - 1 through 15 (of 26 total)

You must be logged in to reply to this topic.