Umlauts in Search lead to JS-Error.

[animaux]

When using the search term schö for example an error is thrown and the search is not successful:

TypeError: undefined is not an object (evaluating 'a.length')

[animaux]

This results in Strange behaviour:

If I try to look for an entry named Hänigsen the above error will prevent a successful search as soon as the »ä« comes into play.

However, if I type nigsen, the entry is succefully found and presented. After that I can even start a new search with the search term Hän… actually working.

I suspect there is some encoding problem happening during typing or search submission. I wonder if @nilshoerrmann has has this problem too? :)

[animaux]

Not sure if the extension or symphony is at fault.

[nilshoerrmann]

No, I haven't encountered this issues yet.

[animaux]

Hmpfdgnf. Danke :)

[animaux]

Apparently the problem seems to be with Umlauts in the first 3 characters of a search term.

[animaux]

Using a reflection field instead of a text field for the association makes me able to find the Umlaut-Entries but still throws the error.

[animaux]

Hmmm. Can’t reproduce the error on a copy of the SOR site trying several versions of AI Editor and the Association field itself. Must be some other extension gretching in.

[nilshoerrmann]

The search is powered by Selectize that is bundled with Symphony (or at least was – I'm not sure how this is handled in the latest releases). Maybe Selectize was updated in the core resulting in compatibility issues?

[animaux]

I have used an updated version of selectize in my fork-set that overrides the bundled one:

https://github.com/animaux/association_ui_selector
https://github.com/animaux/association_ui_editor

Installing those does not break the SOR-Site either … must be something else in play.

[animaux]

Really beats me. I’ve copied all the extensions, symphony-folder and rebuilt the section/link-structure from a site with the error but still can’t reproduce it. ¯_(ツ)_/¯

[nilshoerrmann]

Have you tried another browser?
Have you checked which line is causing the error?

[animaux]

[animaux]

And Firefox Dev:

[animaux]

Not really helping because the files mentioned are identical on two sites. One has the error. One not.

[animaux]

Extensions are identical too. Site structure. I’m struggling to find the difference …

[twiro]

I don't think the problem is caused by the extension itself - I remember having some filtering trouble with umlauts in the publish filtering feature (which also relies on selectize) too. And as far as I can remember I was getting similar javascript-errors as the ones you're posting.

I did some tests back then, compared datasource filtering with publish filtering and noticed that there are really a lot of quirks regarding filtering umlauts and special characters. Regarding selectize I even noticed differences in typing a character and using the proposed autocomplete-values.

You can see the results for the basic input-field here.

I submitted a PR that fixes the filtering-bugs number-field has (will be in 2.7), but didn't dig deeper into the umlaut-issues...

[animaux]

Thanks Roman.

[nilshoerrmann]

That's a great test, Roman.
You should notify @nitriques and @brendo about the results (if you haven't done this already).

[twiro]

That's a great test, Roman.

Thank you! I got inspired by this page in the wiki and thought it would be cool to have this kind of overview for all (core) fields.

You should notify @nitriques and @brendo about the results (if you haven't done this already).

You bet I did - but I guess there are no resources left at the moment to take care of things like these. Though these documented tests would probably be of great help if someone really wanted to start fixing these bugs one by one... as proposed in the post linked above one could start to file separate issues for the separate bugs, but I didn't want to flood symphony's issue tracker with a ton of "edge case issues" that probably won't get fixed for a looong time.

But I do hope to get back to these tests some day, update them and maybe try fixing some of the more problematic issues myself. I also think it would be a great addition to some extensions to directly add these filtering-test-reports to the extension repository (the way I did with slider field) - helps giving a better understanding of what each field's filtering capabilities (and shortcomings/bugs) are.

But for now... enough issue-hijacking ;)

[animaux]

But for now... enough issue-hijacking ;)

No worries, good thing you did.

I’ve checked the DB encoding too. Same.

[animaux]

.htaccess and config.php are also not the problem …

[nitriques]

@animaux curious to see how Chrome handles it. Same error ? I know for a fact that Safari uses UTF-16 in their jscore implementation. But seeing that the error is in jquery itself puzzle me a lot. But again, PHP has not internal encoding (it's just binary data) but you can "format" that binary code to whatever you want.

Are the sites both on the same server ? Same php.ini file ?

[nitriques]

@twiro

but I guess there are no resources left at the moment to take care of things like these.

Yeah, sadly. We got "big" clients this year which means we are doing more maintenance that what we've planned, which reduced my "free" time, both at work and at home. But dealing with encoding always was and still is a pain. I wonder how other systems handle those cases, but from my experience, it's not really better.

[nitriques]

And this is worth reading: http://softwareengineering.stackexchange.com/a/151296

strlen is encoding-agnostic and assumes a fixed-width 8 bit encoding, that is, one byte per character, so it will count bytes, not characters.

Make sure to use General::strlen() and have mb_string installed.

[animaux]

Are the sites both on the same server ? Same php.ini file ?
Yes, absolutely.

And it’s the same in chrome:

[animaux]

I’m not sure entirely it’s an encoding problem since the error comes up as well if I type in at least three standard ascii characters. But in that case the search seems to work.

[animaux]

@jonmifsud’s suggestion helped solve the Umlaut-issue:

change line 12 of association_ui_selector/content/content.query.php to
$search = General::sanitize(urldecode($_GET['query']));

About the error:

in line 215 of the JS there should be a check if result.entries is populated

So that's the basic recipe to fix this problem :)

[animaux]

Fixed in #57