Enable handling of unicode characters

[bewedeg]

Hi Tony,
I think i might have found a small issue in

FixedLengthFileHandler/src/FixedLengthFileReader.php

Line 62 in 3c999d3

$fieldConfig, substr($line, $i, $fieldConfig['size'])

In Germany and other countries there are special characters like ä, ö, ü, ß and so on. If the underlying file has these encoded in unicode substr in PHP does not handle these characters very nicely, as the character bytewise is cut in half leaving garbage. PHP has mb_substr(...) to handle this character in the correct manner.

I replaced that in a local copy and i can confirm that it works as expected.
$fieldConfig, mb_substr($line, $i, $fieldConfig['size'])

Regards,
Bernhard

[tonyfarney]

You are right! The library lacks in providing support for that kind of situation. I am not used with those characters, but it appears to be an encoding issue. In the case you pointed out, my understanding is that a symbol that is considered to be 1 character is actually encoded as 2 characters. Is that the case? If so, I think that the correct approach is to use an encoding that properly represents that simbols as 1 character.

[bewedeg]

I just looked at the file and its plain ANSI

Ü is stored as HEX dc

which leaves me even more puzzled why substr doesn't handle this case correct but mb_substr does.

[tonyfarney]

Interesting. If just replacing by mb_substr solved the issue, please feel free to issue a PR. But it's necessary to check if the function exists (as it is part of the mbstring extension) before using it. If it's not defined, fallback to substr.