C AST output seems totally broken

[cfcommando]

Context: I'm writing a CommonMark -> NSAttributedString parser, so it can be rendered natively on iOS devices (and maybe OS X later).

Am I missing something obvious?

example1.txt:

Normal paragraph


Paragraph 2


foo [bar](http://example.com) baz

Dumped AST:

document
  paragraph
    str "Normal paragraphParagraph 2foo "
    link url="http://example.com" title=""
      str "bar"
    str " baz"

example2.txt:

Normal paragraph


1. one
2. two
3. three


foo [bar](http://example.com) baz

Dumped AST:

document
  paragraph
    str "Normal paragraph"
  list (type=ordered tight=true start=1 delim=period)
    list_item
      paragraph
        str "one"
    list_item
      paragraph
        str "two"
    list_item
      paragraph
        str "threefoo "
        link url="http://example.com" title=""
          str "bar"
        str " baz"

Parsing code, cribbed from main.c:

    NSMutableAttributedString *attributedString = [[NSMutableAttributedString alloc] init];

    __block block *cursor = make_document();
    __block int line_number = 1;
    [string enumerateLinesUsingBlock:^(NSString *line, BOOL *stop) {
        const char * c_line = [line UTF8String];
        const bstring b_line = bfromcstr(c_line);

        int result = incorporate_line(b_line, line_number, &cursor);
        NSAssert(result == 0, @"error incorporating line %d", line_number);

        bdestroy(b_line);
        line_number++;
    }];

    while (cursor != cursor->top) {
        finalize(cursor, line_number);
        cursor = cursor->parent;
    }
    NSParameterAssert(cursor == cursor->top);

    finalize(cursor, line_number);
    process_inlines(cursor, cursor->attributes.refmap);

#ifdef DEBUG
    print_blocks(cursor, 0);
#endif

    concatenate_blocks(cursor, attributedString);

    free_blocks(cursor);

(returns attributedString)

(side note: bstring is going to be a huge waste of / terrible for memory/performance)

[cfcommando]

Turns out that when iterating over lines, incorporate_line implicitly depends on the newline not being stripped out. The workaround is as simple as: bconchar(b_line, '\n');

[jgm]

+++ cfcommando [Sep 11 14 13:37 ]:

Context: I'm writing a CommonMark -> NSAttributedString parser, so it can be rendered natively on iOS devices (and maybe OS X later).

Am I missing something obvious?

example1.txt:

Normal paragraph

Paragraph 2

foo [bar](http://example.com) baz

Dumped AST:

document
 paragraph
   str "Normal paragraphParagraph 2foo "
   link url="http://example.com" title=""
     str "bar"
   str " baz"

I don't know what happened here. I tried your input and got:

foo [bar](http://example.com) baz
document
  paragraph
    str "Normal paragraph"
  paragraph
    str "Paragraph 2"
  paragraph
    str "foo "
    link url="http://example.com" title=""
      str "bar"
    str " baz"

which looks fine. Are you perhaps using CR as line ending?

(side note: bstring is going to be a huge waste of / terrible for memory/performance)

I know. @vmg is currently rewriting the C code for performance,
and the first thing he did was get rid of bstring. There will be
huge speedups!

[cfcommando]

Nope, not CR. I figured out that enumerating over lines as I did automatically stripped out the newlines, but incorporate_line expects them to be left in.