Skip to content

Parse flatfile with 0x00 null separator #49

@ajhalls

Description

@ajhalls

I have a flatfile that uses fixed width columns, has 0x00 (hex null) separators but from what I can see, that also have null padding that I am trying to query.
Because the incoming flatfile doesn't have column headers, this looks like it would be a great tool to work with, but wasn't sure if it was possible to use a hex separator?

Here is an example of the data where � = null:

Ashlyn         �Vandiver       �Ashlyn         �    �4466743����85344
Bill               �Vandiver       �Bill               �    �4466744����85344

(this binary isn't the same as above, , just a representation)
94 DE 94 DE 86 03 64 00 65 00 D1 42 00 00 4A 6F 61 6E 6E 65 20 20 20 20 20 20 20 20 20 00 41 62 62 65 79 20 20 20 20 20 20 20 20 20 20 00 4A 6F 61 6E 6E
94 DE 94 DE 93 03 64 00 65 00 01 B6 03 00 4E 6F 72 6D 61 6E 20 20 20 20 20 20 20 20 20 00 45 61 74 6F 6E 20 20 20 20 20 20 20 20 20 20 00 4E 6F 72 6D 61
94 DE 94 DE EB 03 64 00 65 00 A9 6E 02 00 44 61 6E 69 65 6C 20 20 20 20 20 20 20 20 20 00 43 61 67 65 20 20 20 20 20 20 20 20 20 20 20 00 44 61 6E 69 65
94 DE 94 DE 85 03 64 00 65 00 D1 07 00 00 4A 75 6C 69 65 20 20 20 20 20 20 20 20 20 20 00 51 75 69 6E 63 79 20 20 20 20 20 20 20 20 20 00 4A 75 6C 69 65
5C D9 94 DE 94 03 64 00 65 00 0A 07 00 00 4D 61 72 79 20 20 20 20 20 20 20 20 20 20 20 00 43 61 75 6C 6B 20 20 20 20 20 20 20 20 20 20 00 4D 61 72 79 20

It looks like strings are padded with spaces, and integer values with null. I started writing some logic to play with trying to convert it to CSV, but I will have to do some magic with datatypes that you have already accomplished since I have byte groups that are the dates and integers stored in little endian.

                
for (int i = 0; i < block.Length; i++) //one fixed width line from flatfile
{                     
    if (i == block.Length)
    {
        line += "\n"; // this is the end of the line
    }

    if (i > 1 && i< block.Length)
    {
        // deal with null padding and create csv
        if (block[i] == 0x00 && block[i]+1 != 0x00)
        {
            line += "','";
        }else if (block[i] == 0x00 && block[i]-1 != 0x00)
        {
            line += "','";
        }
    }

    if (block[i]!=0x00)
    {
        line += (char)block[i]; // it isn't null, so display it
    }
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions