help with storing large text fields

  • dnikola
    Participant
    # 6 months, 1 week ago

    Hello there,
    When I try to index custom post which has some meta fields with large amount of transcript text I got an error
    Solr HTTP error: OK (400) { “responseHeader”:{ “status”:400, “QTime”:95}, “error”:{ “metadata”:[ “error-class”,”org.apache.solr.common.SolrException”, “root-error-class”,”org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException”], “msg”:”Exception writing document id 17562 to the index; possible analysis error: Document contains at least one immense term in field=\”_transkript_str\” (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: ‘[90, 97, 115, 105, 106, 101, 100, 97, 110, 106, 101, 32, 80, 114, 118, 111, 103, 32, 107, 111, 110, 103, 114, 101, 115, 97, 32, 74, 101, 100]…’, original message: bytes can be at most 32766 in length; got 128127. Perhaps the document has an indexed string field (solr.StrField) which is too large”, “code”:400}}

    How I can resolve this issue?

    Regards!

    wpsolr
    Keymaster
    # 6 months, 1 week ago

    Hello,

    Your custom field “transkript” is setup with a Solr field type that cannot store more than 32K characters.

    You need to change your field’s type in screen 2.2, and select instead the large text type “Text, if too long for other text types (cannot be used as facet)” in the dropdown list.

    Do not forget to reindex all your data afterwards.

    Change field type to large text

    dnikola
    Participant
    # 5 months, 3 weeks ago

    Hi,

    thanks for your replay, and sorry for my late one.

    Ok, from beginning it was setup like you suggested please checkout this picture

    https://prntscr.com/mctmi0

    i tried to “confirm” settings and to re-index but again getting same error

    wpsolr
    Keymaster
    # 5 months, 3 weeks ago

    How do you fill your custom field with a string of length greater than 32K?

    dnikola
    Participant
    # 5 months, 2 weeks ago

    you asked for this info

    https://prntscr.com/mdddrh

    I took a meta field pasted it in notepad ++ and got this value ๐Ÿ™‚

    wpsolr
    Keymaster
    # 5 months, 2 weeks ago

    THanks.

    I could reproduce your issue in my environment, and fixed it. I also added the automated tests to prevent the bug coming back in the future. It will be delivered with the next release.

    If you want the fix now, let me know.

    dnikola
    Participant
    # 5 months, 2 weeks ago

    Yes please if possible ๐Ÿ™‚

    Thanks!

    wpsolr
    Keymaster
    # 5 months, 2 weeks ago

    Please replace WPSOLR with https://www.dropbox.com/s/dw5vqnr2j2q4ger/wpsolr-pro.zip?dl=0 (this link is valid for one hour)

    It should prevent the “immense term” error, for Solr and Elasticsearch.

    dnikola
    Participant
    # 5 months, 2 weeks ago

    I am not getting notifications about replays but I was able to get this file from link even after more than 27h (just to know ๐Ÿ˜› )

    i activated this version but getting 500 error

    i was able to get this from debug log

    PHP Parse error: syntax error, unexpected end of file in /home/*******/httpdocs/wp-content/plugins/wpsolr-pro/wpsolr/pro/extensions/theme/layout/ion_range_slider/class-wpsolr-ui-layout-ion-range-slider.php on line 228

    wpsolr
    Keymaster
    # 5 months, 2 weeks ago

    What is your OS, and PHP version?

    wpsolr
    Keymaster
    # 5 months, 2 weeks ago

    It appears it could be a short tag in the file (line 40). I will disable short tags during my automated tests to prevent this error ever happen again.

    dnikola
    Participant
    # 5 months, 2 weeks ago

    Centos 7 php7.0 on this test server

    should i download again from same link?

    Thanks!

    wpsolr
    Keymaster
    # 5 months, 2 weeks ago

    Unfortunately, I cannot generate a new fix at the moment.

    You can just edit the file /home/*******/httpdocs/wp-content/plugins/wpsolr-pro/wpsolr/pro/extensions/theme/layout/ion_range_slider/class-wpsolr-ui-layout-ion-range-slider.php, and replace <? with <?php at line 40.

    Or enable php short tags.

    The fix will come officially in release 21.3.

    dnikola
    Participant
    # 5 months, 2 weeks ago

    Hi

    thanks!

    That fixed a problem ๐Ÿ™‚

    Regards!

Viewing 14 posts - 1 through 14 (of 14 total)

You must be logged in to reply to this topic.