[maxwatson]

4 years, 1 month ago #18723

Hello, any ideas how can I remove this limit when using WPSOLR with Elasticsearch? My store has 150K products and when I build the index, I can only see the most recent 10K indexed products.
Thanks!

[wpsolr]

4 years, 1 month ago #18724

Do you mean like https://stackoverflow.com/questions/41655913/elk-how-do-i-retrieve-more-than-10000-results-events-in-elastic-search ?

[maxwatson]

4 years, 1 month ago #18725

I see they recommend here using ‘scroll’, but how and where can I apply this to WPSOLR?

[wpsolr]

4 years, 1 month ago #18726

I added Elasticsearch scroll in the latest beta release, to be able to generate huge Yoast sitemaps (million+).

What would be your need?

[maxwatson]

4 years, 1 month ago #18727

Currently I have WPSOLR PRO 21.8, how can I get the beta release?
The problem is that the category products are limited to 10K, which is a problem for bigger categories.
Also I think facets filtering is affected by this also, because some terms don’t appear in filters.

[wpsolr]

4 years, 1 month ago #18729

The beta is for sitemaps only, because of a limitation of Google’s maximum number of files in a sitemaps (less files means more links in a file, sometimes more than 10K).
But for category products, you should have a pagination to limit the number of results fetched.

For facets, they should not be affected by the number of results. Clients’ sites with 500K products or a million listings can display all facets. Perhaps you can increase the option “No. of values to be displayed by filters” in screen 2.1?

[maxwatson]

4 years, 1 month ago #18730

Yes, for categories I already have pagination activated, 12 products per page.

[maxwatson]

4 years, 1 month ago #18734

Also the weird thing is that I’ve set No. of values to be displayed by filters: 0 for unlimited values and I’m still getting 10 results in sidebar filters.
I’ve replaced front-end archives for: Search Home/Blog Categories Tags Post types

[maxwatson]

4 years, 1 month ago #18735

In error.log I’m getting this error:

[error] 22638#0: *2675819 FastCGI sent in stderr: “PHP message: WPSOLR message: {“error”:{“root_cause”:[{“type”:”illegal_argument_exception”,”reason”:”Result window is too large, from + size must be less than or equal to: [10000] but was [17528]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.”}],”type”:”search_phase_execution_exception”,”reason”:”all shards failed”,”phase”:”query”,”grouped”:true,”failed_shards”:[{“shard”:0,”index”:”XXX”,”node”:”fwq0djM1XXXizUMgOkT3A”,”reason”:{“type”:”illegal_argument_exception”,”reason”:”Result window is too large, from + size must be less than or equal to: [10000] but was [17528]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.”}}],”caused_by”:{“type”:”illegal_argument_exception”,”reason”:”Result window is too large, from + size must be less than or equal to: [10000] but was [17528]. See the scro…PHP message: WPSOLR trace:

[maxwatson]

4 years, 1 month ago #18737

For example if I access directly the search url like this:

index_name/_search?scroll=1m

I get
“hits” : {
“total” : {
“value” : 195382,
“relation” : “eq”
},

Without scroll=1m parameter I get

“hits” : {
“total” : {
“value” : 10000,
“relation” : “gte”
},

So how can I add this scroll parameter to the plugin?

[wpsolr]

4 years, 1 month ago #18738

I do not understand why you get so many results on one single query. The pagination size in screen 2.2 should limit the results to 20 or something around.

[maxwatson]

4 years, 1 month ago #18773

This limit appears only on store categories (or other pages like brands or product tags) with over 10K products.
Pagination from tab 2.1 works perfect, if I set 12 or 48 products per page, I get that number on the front also.
For search results I didn’t notice this limitation because until now I didn’t have such a big number of results.

Also in tab 2.1 Replace front-end archives I’ve checked only Search, but it applies the query to product categories also, is this normal?
I see Warning: permalinks must be activated, but I don’t know where to activate them.

I’ve tried modifying index.max_result_window in ES to 500000, but nothing changes.

[wpsolr]

4 years, 1 month ago #18778

“Also in tab 2.1 Replace front-end archives I’ve checked only Search, but it applies the query to product categories also, is this normal?”
Because “Replace categories search” is selected on the WPSOLR WooCommerce extension?

[maxwatson]

4 years, 1 month ago #18785

Yes, this is true, I’ve unchecked Replace WooCommerce category and shop search with WPSOLR’s but now I have some new problems.
Now the sidebar WPSOLR facets are all gone. This is not very good, specially because I’ve had there the price filter from WPSOLR and it was really cool.
Also the sort by items don’t work as expected. I have sort by _sale_price_ascending, _sale_price_descending, newest and oldest, when I try to sort by something, nothing happens.

As soon as I check Replace WooCommerce category and shop search with WPSOLR’s, the sidebar facets appear in product categories and sorting also works like a charm. But also the 10K limit is back in place and I can only get 10K products in those categories.

I think the best solution is to leave it checked, but what can be done about this awful 10K limit?

[dnikola]

4 years ago #18806

I am interested in same question.
I am trying to replace wp admin post edit page on site with 600 000 + posts, but i can see only 10000 when I open All posts page https://prntscr.com/ro2kaz