[m-users.] Function to remove unicode byte order mark
novalazy at gmail.com
Sun Feb 26 11:22:32 AEDT 2017
On Sat, 25 Feb 2017 08:57:58 +0100, Dirk Ziegemeyer <dirk at ziegemeyer.de> wrote:
> I’d like to share a litte piece of code that removes the byte order mark (BOM) from the top of an utf-8 file.
> I didn’t even know that a byte order mark exists until I saved an Excel table as Unicode.txt in order to be able to read it with Mercury. BOM is preserved during conversion to utf-8.
> As Mercury doesn’t remove the BOM by itself, it’s up to the application to deal with it.
> % Remove optional unicode byte order mark (BOM) from the beginning
> % of an utf-8 file
> :- func remove_byte_order_mark(string) = string.
> remove_byte_order_mark(RawFirstLine) = FirstLine :-
> ( if
> string.first_char(RawFirstLine, FirstChar, Rest),
> char.to_int(FirstChar) = 0xfeff % Unicode byte order mark (BOM)
> then FirstLine = Rest
> else FirstLine = RawFirstLine
A slight improvement is to use the (in, in, uo) mode of first_char to
avoid copying the rest of the string unless the first char is the BOM,
first_char(S, '\xFEFF\', Rest)
More information about the users