As you activated the “Display acf field names on filters” option on the WPSOLR ACF add-on, WPSOLR is retrieving ACF field. Unfortunately, there is no other way to do so but to get fields from the post_meta table.
You could eventually cache the query results with a cache like Redis or memcached?