Elasticsearch NoVM not working - Search failed with the following reason: failed to create query


#1

I managed to get the autosuggest work by using the elasticsearch.yml config from VM instance but:
using search (hitting the button) end up with:

Spryker\Client\Search\Exception\SearchResponseException - Search failed with the following reason: failed to create query: { "bool" : { "must" : [ { "match" : { "store" : { "query" : "DE", "operator" : "OR", "prefix_length" : 0, "max_expansions" : 50, "fuzzy_transpositions" : true, "lenient" : false, "zero_terms_query" : "NONE", "boost" : 1.0 } } }, { "match" : { "locale" : { "query" : "en_US", "operator" : "OR", "prefix_length" : 0, "max_expansions" : 50, "fuzzy_transpositions" : true, "lenient" : false, "zero_terms_query" : "NONE", "boost" : 1.0 } } } ], "filter" : [ { "nested" : { "query" : { "bool" : { "filter" : [ { "term" : { "string-facet.facet-name" : { "value" : "label", "boost" : 1.0 } } }, { "term" : { "string-facet.facet-value" : { "value" : 4, "boost" : 1.0 } } } ], "disable_coord" : false, "adjust_pure_negative" : true, "boost" : 1.0 } }, "path" : "string-facet", "ignore_unmapped" : false, "score_mode" : "avg", "boost" : 1.0 } }, { "nested" : { "query" : { "term" : { "integer-facet.facet-name" : { "value" : "price-DEFAULT-EUR-GROSS_MODE", "boost" : 1.0 } } }, "path" : "integer-facet", "ignore_unmapped" : false, "score_mode" : "avg", "boost" : 1.0 } } ], "disable_coord" : false, "adjust_pure_negative" : true, "boost" : 1.0 } } [index: index_page] [reason: all shards failed]. Query: {"query":{"bool":{"filter":[{"nested":{"query":{"bool":{"filter":[{"term":{"string-facet.facet-name":{"value":"label","boost":1}}},{"term":{"string-facet.facet-value":{"value":4,"boost":1}}}]}},"path":"string-facet"}},{"nested":{"query":{"term":{"integer-facet.facet-name":{"value":"price-DEFAULT-EUR-GROSS_MODE","boost":1}}},"path":"integer-facet"}}],"must":[{"match":{"store":"DE"}},{"match":{"locale":"en_US"}}]}},"_source":["search-result-data"],"from":0,"size":12,"aggs":{"category.all-parents.category":{"terms":{"field":"category.all-parents","size":1000}},"integer-facet":{"nested":{"path":"integer-facet"},"aggs":{"integer-facet-name":{"terms":{"field":"integer-facet.facet-name","size":10},"aggs":{"integer-facet-stats":{"stats":{"field":"integer-facet.facet-value"}}}}}},"string-facet":{"nested":{"path":"string-facet"},"aggs":{"string-facet-name":{"terms":{"field":"string-facet.facet-name","size":10},"aggs":{"string-facet-value":{"terms":{"field":"string-facet.facet-value"}}}}}}}}

trying to enter product page also ens up with elasticsearch exception.


#2

Hi @janmyszkier,

I’ve run your query on my local VM and I get back results correctly without any exception. Are you using the VM provided by Spryker as well? Make sure that you are using ES 5.x (preferably 5.6) version.

I’ve noticed that you are querying “index_page” index. If it’s intentional, then make sure that you have this index set up correctly by checking its mapping by runnin the GET index_page/_mapping query.

Best regards,
Tamás


#3

not that I’m running the VM, but I had to run it eventually to figure out what docs were still missing in the installation process, that’s where I found spryker actully modified elasticseach config in the VM but didn’t mention
it in the docs.

the GET query returns the following:

{"index_page":{"mappings":{"page":{"properties":{"category":{"properties":{"all-parents":{"type":"long"},"direct-parents":{"type":"long"}}},"completion-terms":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}},"fielddata":true},"full-text":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"full-text-boosted":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"integer-facet":{"properties":{"facet-name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"facet-value":{"type":"long"}}},"integer-sort":{"properties":{"category:1":{"type":"long"},"category:10":{"type":"long"},"category:11":{"type":"long"},"category:12":{"type":"long"},"category:13":{"type":"long"},"category:14":{"type":"long"},"category:15":{"type":"long"},"category:2":{"type":"long"},"category:3":{"type":"long"},"category:4":{"type":"long"},"category:5":{"type":"long"},"category:6":{"type":"long"},"category:7":{"type":"long"},"category:8":{"type":"long"},"category:9":{"type":"long"},"price":{"type":"long"},"price-DEFAULT-CHF-GROSS_MODE":{"type":"long"},"price-DEFAULT-CHF-NET_MODE":{"type":"long"},"price-DEFAULT-EUR-GROSS_MODE":{"type":"long"},"price-DEFAULT-EUR-NET_MODE":{"type":"long"},"price-ORIGINAL-CHF-GROSS_MODE":{"type":"long"},"price-ORIGINAL-CHF-NET_MODE":{"type":"long"},"price-ORIGINAL-EUR-GROSS_MODE":{"type":"long"},"price-ORIGINAL-EUR-NET_MODE":{"type":"long"},"rating":{"type":"long"},"weight":{"type":"long"}}},"is-active":{"type":"boolean"},"locale":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"product-lists":{"properties":{"blacklists":{"type":"long"},"whitelists":{"type":"long"}}},"search-result-data":{"properties":{"abstract_name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"abstract_sku":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"description":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"id_category":{"type":"long"},"id_cms_page":{"type":"long"},"id_product_abstract":{"type":"long"},"id_product_abstracts":{"type":"long"},"id_product_labels":{"type":"long"},"id_product_set":{"type":"long"},"image_sets":{"properties":{"default":{"properties":{"external_url_large":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"external_url_small":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"id_product_image":{"type":"long"}}}}},"images":{"properties":{"created_at":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"external_url_large":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"external_url_small":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fk_product_image":{"type":"long"},"fk_product_image_set":{"type":"long"},"id_product_image":{"type":"long"},"id_product_image_set_to_product_image":{"type":"long"},"sort_order":{"type":"long"},"updated_at":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"is_active":{"type":"boolean"},"meta_description":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"meta_keywords":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"meta_title":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"price":{"type":"long"},"prices":{"properties":{"CHF":{"properties":{"GROSS_MODE":{"properties":{"DEFAULT":{"type":"long"},"ORIGINAL":{"type":"long"}}},"NET_MODE":{"properties":{"DEFAULT":{"type":"long"},"ORIGINAL":{"type":"long"}}}}},"EUR":{"properties":{"GROSS_MODE":{"properties":{"DEFAULT":{"type":"long"},"ORIGINAL":{"type":"long"}}},"NET_MODE":{"properties":{"DEFAULT":{"type":"long"},"ORIGINAL":{"type":"long"}}}}}}},"product_set_key":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"rating":{"type":"float"},"review_count":{"type":"long"},"type":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"url":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"weight":{"type":"long"}}},"store":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"string-facet":{"properties":{"facet-name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"facet-value":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"string-sort":{"properties":{"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"suggestion-terms":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"type":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}},"fielddata":true}}},"product-review":{"properties":{"created-at":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"id-product-abstract":{"type":"long"},"rating":{"type":"long"},"search-result-data":{"properties":{"created_at":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"customer_reference":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"description":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fk_locale":{"type":"long"},"fk_product_abstract":{"type":"long"},"id_product_review":{"type":"long"},"nickname":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"rating":{"type":"float"},"status":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"summary":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"updated_at":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"store":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}}}

but even If it doesn’t match whatever Spryker expects (which is clear from the search errors I’m getting) I have no idea how to tell spryker to create the mapping it expects. The mapping data I currently have was created by vendor/bin/install. What should I run to create the ES mappings correctly right after vendor/bin/install?


#4

So my assumption is that you’ve changed the environment configurations to use index_page name as index but you missed to align it with the search setup. You can get info about how to set up ES index for Spryker in: https://documentation.spryker.com/content/capabilities/search_and_filter/search-configure-elasticsearch.htm.

The search installer is looking for JSON files described in the document above. In order to have index_page index you should have a store called “index” and a file called “page.json” with the mapping in it, but I doubt that this is the case.

You can check what indexes you have by the GET _mapping query and you’ll see there something like “de_search” ([storeName]_[jsonFileName]) depending on your store configuration.


#5

Hi Jan,

I guess it might be the result of wrong order. In Elasticsearch you have to create mapping first, on an empty index. Otherwise, if you feed data to Elasticsearch, it will assume default mappings. The tricky part of Elasticsearch logic is that you can’t change mapping afterwards. So it’s important to create mapping first and then feed it with the data.
I would recommend following steps:

  • stop Jenkins in order to make sure that no workers are running and nothing is writing to Elasticsearch
  • drop all Elasticsearch indices
  • run the installer - it will create indexes, mappings and then push data into it

The suggestion above is just a guess as I didn’t reproduce your problem on my machine, but it’s worth a try.

Best regards,
Marek Obuchowicz
KoreKontrol.eu - Spryker hosting company


#6

well, according to the output of the vendor/bin/install jenkins is already stopped before installation takes place.
anyway I tried what you suggested and it still doesn’t work.

  1. disable jenkins:
$ vendor/bin/console setup:jenkins:disable"
Store: DE | Environment: development
Jenkins disabled (response: 302)
  1. check existing indexes:
curl -XGET spryker-elasticsearch:9200/_cat/indices
  1. remove indexes:
curl -XDELETE spryker-elasticsearch:9200/us_search
curl -XDELETE spryker-elasticsearch:9200/de_search
curl -XDELETE spryker-elasticsearch:9200/at_search
curl -XDELETE spryker-elasticsearch:9200/index_page
  1. confirm their removal with
curl -XGET spryker-elasticsearch:9200/_cat/indices

re-run installer
5. run /vendor/bin/install

same result.
not to mention: removing production database just to restore correct elasticsearch indexes/mappings is a silly idea. there should be another way to regenerate them.


#7

Ok, then it’s something different, would check on your end suggestion from Tamas.

Please note that the installer script is on purpose called development.yml and is supposed to run on development environment only. It is definitely not a recipe that should be called on production. You can read the development.yml file to check which vendor/bin/console actions are triggered and, based on your setup details and requirements, it your/hoster job to ensure that you execute proper steps in production environment. Those would definitely not exclude dropping the database.


#8

What Tamas suggested is to read the doc which only tells you to run search:setup (and that thing is already run by the installer:

Command setup-search for DE store [vendor/bin/console setup:search] (0.45s)
Command setup-search for AT store [vendor/bin/console setup:search] (0.44s)
Command setup-search for US store [vendor/bin/console setup:search] (0.45s)

so… it doesn’t help either.


#9

You might have missed my point so let me try to ask in different way. Where is the index_page index coming from and why is it not set up correctly?


#10

I hoped it would be clear from my first post this is the question I’m asking you :wink:
I followed your install NoVM docs and run official installer script and I have this error.
so let me quote you on your quesiton:
“Why is it not set up correctly?”
Why Spryker itself cannot setup ES the way it requires it to be?


#11

Alright, it looks like there is an error in the installation guide which says to set the index name to index_page. This is wrong, please set the configuration of the index name to de_search for DE store, at_search for AT store, etc.