Edi Reader does not respect X12 advised control characters

[cleftheris]

In all known EDI formats there are default control characters these are documented in the default EdiGrammar presets.

• EdiGrammar.NewEdiFact()
• EdiGrammar.NewX12()
• EdiGrammar.NewTradacoms()

Now in the EdiFact and X12 cases there is a way for the transmitter (he who sends the edi transmission)
to advise differently for each of the control characters.

Control characters are considered to be the segment, element & component seperators plus the escape character.

In X12 the current implementation completely ignores them.

[cleftheris]

@raholland79 A temp workaround until this is fixed would be to alter the default grammar with different presets:

public class CustomX12Grammar : EdiGrammar
{
    public CustomX12Grammar : base() 
    {
            _ComponentDataElementSeparator = '>',
            _DataElementSeparator = '*',
            _DecimalMark = '.',
            _ReleaseCharacter = null,
            _Reserved = new char[0],
            _SegmentTerminator = '~',
            _ServiceStringAdviceTag = null,
            _InterchangeHeaderTag = "ISA",
            _FunctionalGroupHeaderTag = "GS",
           _MessageHeaderTag = "ST",
            _MessageTrailerTag = "SE",
            _FunctionalGroupTrailerTag = "GE",
            _InterchangeTrailerTag = "IEA",
    }
}

[raholland79]

So in the case of

ISA_00_ 00 _02_SCAC _ZZ_MGCTLYST 160726_0836_U_00400_000002356_0_T>

We have no segment terminator - How do I represent that?

Also using the example given, on every file I try, I get this error

Invalid character after parsing segment name. Expected '*' but got: *. Path '', line 1, position 3.

[cleftheris]

sorry about that. My intention was to show how you could work around this by testing different character sets for the delimiters. Since it is not clear without the complete edi transmition what they are.

Consider the above as the default configuration.

I am guessing that a space is used as the DataElementSeparator = ' ' and the > as _SegmentTerminator = '>' for the ComponentDataElementSeparator I have no idea.

[cleftheris]

take a look this is what I found about what X12 990 looks like. Its a sample I found googling about it

ISA*00*          *00*          *12*4405197800     *01*999999999      *111219*1742*U*00400*000000003*0*P*>

[cleftheris]

Ok according to these guys here

EDI data is exchanged in text files. With this in mind, EDI standards recommendations do not result in a viewable file, and common industry practice produces corrupted files when collisions occur between data and delimiters.

this means that there is a great possibility that your delimiters are not visible.

According to the same link standards suggest you use the following control characters (some of them invisible). Try this:

public class CustomX12Grammar : EdiGrammar
{
    public CustomX12Grammar : base() 
    {
           _ComponentDataElementSeparator = '>'; // for 990  
           //_ComponentDataElementSeparator = ':'; // for your 214 
           _SegmentNameDelimiter = (char)Int16("001D", System.Globalization.NumberStyles.AllowHexSpecifier);
           _DataElementSeparator = (char)Int16("001D", System.Globalization.NumberStyles.AllowHexSpecifier);
           _SegmentTerminator = (char)Int16("001C", System.Globalization.NumberStyles.AllowHexSpecifier);
            _DecimalMark = '.',
            _ReleaseCharacter = null,
            _Reserved = new char[0],
            _ServiceStringAdviceTag = null,
            _InterchangeHeaderTag = "ISA",
            _FunctionalGroupHeaderTag = "GS",
           _MessageHeaderTag = "ST",
            _MessageTrailerTag = "SE",
            _FunctionalGroupTrailerTag = "GE",
            _InterchangeTrailerTag = "IEA",
    }
}

PS: You could try to see the Unicode char by reading the file to text and debugging in visual studio to make sure.

[raholland79]

That matches mine exactly'

However, using the default config above for files that conform to the default configuration I still get:

Invalid character after parsing segment name. Expected '*' but got: *. Path '', line 1, position 3.

The code is the default as you've written above - and the 214 works fine with the default NewX12()

But using the custom grammar gives the expect * but got * error.

var customGrammar = new CustomX12Grammar();
using (var stream = new StreamReader(@"C:\sandbox\drdispatch\EDI\assets\214-MGCTLYST-SAMPLE.EDI"))
    var interchange = new EdiSerializer().Deserialize<Mg214Interchange>(stream, customGrammar);

[raholland79]

I can't get it working. I may have to try Edi Fabric - but would rather not. I have a customer wanting this done NOW though.

[cleftheris]

@raholland79 I can see if I can make this work but I need you to send me the original sample file (not tampered with), If this is sensitive information involved send me via email here: c.leftheris at live.com . I need the originals because any file edited with a text editor may have stripped out the invisible control characters we are searching for.

I have been working hard to release a first version that includes the Serialization (writing back to EDI) for a while now. Not an easy task since this is an opensource project I work on my free time. That said it just came out yesterday v1.1 and its out of the way.

[raholland79]

Oh I know you're working hard - And I appreciate it immensely

I'd be willing to toss some cash your way for your hard work.

I've attached a zip with all 4 files.

If I remember right only the 214 is working out of the box.

EdiSampleFiles.zip

[raholland79]

204 and 990 are top priority

[cleftheris]

@raholland79 Ok the issue here is that the new line \n is used as the special character for Segment Termination. This is considered as whitespace throughout the EdiTextReader and will probably take a day or two to figure out. In the meantime you could replace \n with another valid character like ~ before feeding the stream to Edi.Net and everything should work fine (provided you have the rest of the seperators setup correctly).

[raholland79]

So basically replace the trailing \n on the ISA line in code with a ~ and process from there?

[cleftheris]

No you must replace every occurrence of the \n character with ~.
This will do the trick until this is fixed,

        [Fact]
        public void X12_204_Test() {
            var grammar = EdiGrammar.NewX12();
            grammar.SetAdvice(
                segmentNameDelimiter: '*', 
                dataElementSeparator: '*', 
                componentDataElementSeparator: ':', 
                segmentTerminator: '~', 
                releaseCharacter: null, 
                reserved: null, 
                decimalMark: '.');

            string text = File.ReadAllText(@"C:\sandbox\drdispatch\EDI\assets\204-MGCTLYST-SAMPLE.EDI");
            var segmentCount = 0;
            using (var stream = new MemoryStream(Encoding.UTF8.GetBytes(text.Replace('\n', '~')))) {
                interchange = new EdiSerializer().Deserialize<Models.Transportation_204>(new StreamReader(stream), grammar);
            }
        }

this reads the whole file into a string. Then replaces the character and at last it feeds the result to the EdiSerializer.

[raholland79]

My idiot vendor now says the terminator should be ~ on all files - I'm reconfirming to be sure - but If so this makes the library work as is without any work around and you can backlog this particular bug for more important work.

I'll let you know ASAP.

Again, thanks for the wonderful hard work and excellent communication.

[raholland79]

They're good with ~ so - while this may be a bug I don't need it handled.

[cleftheris]

@raholland79 glad you made it work. Issue #24 is still one of my top priorities.

Thanks.

[PGP-Protector]

I did find a way to solve this issue (Not nice ,but works)
Was having an issue with some FedEx X12 214 Files.
The EdiGrammar.NewX12(); would default to ISA16 = '~' only
And wouldn't actually read the Character in that field '!' (Why???)
Each time it would complain that there's too many characters.

My solution ? Not nice but solves the
Invalid character after parsing segment name. Expected '*' but got: *. Path '', line 1, position 3.
In using the above example code.
Created a New Class Grammer:IEdiGrammer and just used the original EDIGrammer code as a base, but with the values I needed. (Dropped the unused sections also)
FedExX12214.zip

File attached shows the method I did.

Better solution would be to allow proper reading of the ISA16 field though.

[cleftheris]

Hi @PGP-Protector,

Unfortunately this is still an issue. The main reason is that I did not find the time. Changing the code to be able to treat \n (#24) was not trivial and that has stalled this issue as well.

Regarding your workaround, it seems fine to me. The only improvement (suggestion) would be to use the SetAdvice method that can override the characters without you implementing the whole thing.

var grammar = EdiGrammar.NewX12();
grammar.SetAdvice(
                segmentNameDelimiter: '*', 
                dataElementSeparator: '*', 
                componentDataElementSeparator: ':', 
                segmentTerminator: '!', 
                releaseCharacter: null, 
                reserved: null, 
                decimalMark: '.');