External instance filtering with Enketo and ODK Aggregate

[sjudeng]

Hello. Thanks for your work on Enketo and in the XForm community.

I'm having trouble getting a form that I can upload through ODK aggregate that will allow external instance filtering (e.g. search() equivalent) within Enketo. Is this supported?

I'm using the latest fixed version of pyxform to convert the form. This is working for loading data ... pyxform adds the necessary instance element, <instance id="detaildata" src="jr://file-csv/detaildata.csv"/>, and I can upload the form through ODK Aggregate and view it in Enketo. I've stepped through enketo-core:Form-model.js in the web console and verified that the external data is being read and parsed as expected.

The problem is when I try to filter/use the external data in a <select1> question. I've tried following this link, including the hacks to try to fake out ODK Validate. But everything I've tried leads to the same error when I try to upload the form to ODK Aggregate.

HTTP Status 400 - Problem with JavaRosa Parsing Form: org.opendatakit.aggregate.exception.ODKIncompleteSubmissionData: Javarosa failed to construct a FormDef. Is this an XForm definition?

Before I keep looking at it I wanted to get advice on whether the approach is expected to work at all. I know the above link and related discussion on odk-dev are proposals, but it also looked like there was some implementation already in place for this on the Enketo side.

[MartijnR]

Hi Sjudeng,

I think the problem is that in the proposal I initially proposed to use "select_one_external" but then found out that somebody else had already implemented this exact syntax in pyxform (for an awful solution from an XForm perspective, almost as awful as the search() appearance).

So the final solution, which is currently available in the latest pyxform is select_one_from_file detaildata.csv or select_multiple_from_file detaildata.csv. You can put any valid XPath expression in the choice-filter (note that starts-with() and contains() won't pass ODK Validate, but are actually supported in Enketo)

Sorry for the confusion. If it's not working, please let me know and link to the XLSForm and XForm please.

[sjudeng]

Hi Martijn,

Thank you for your quick response.

I'm not able to get this to work, but I'm also new to X[LS]Form. The XML for the simple test I'm trying is below and the XLSForm is here.

<?xml version="1.0"?>
<h:html xmlns="http://www.w3.org/2002/xforms" xmlns:ev="http://www.w3.org/2001/xml-events" xmlns:h="http://www.w3.org/1999/xhtml" xmlns:jr="http://openrosa.org/javarosa" xmlns:orx="http://openrosa.org/xforms" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <h:head>
    <h:title>select from file test</h:title>
    <model>
      <instance>
        <select_from_file_test id="select_from_file_test">
          <state/>
          <meta>
            <instanceID/>
          </meta>
        </select_from_file_test>
      </instance>
      <instance id="cities" src="jr://file-csv/cities.csv">
        <root>
          <item>
            <name/>
            <label/>
          </item>
        </root>
      </instance>
      <bind nodeset="/select_from_file_test/state" type="string"/>
      <bind calculate="concat('uuid:', uuid())" nodeset="/select_from_file_test/meta/instanceID" readonly="true()" type="string"/>
    </model>
  </h:head>
  <h:body>
    <select1 ref="/select_from_file_test/state">
      <label>state</label>
      <itemset nodeset="instance('cities')/root/item">
        <value ref="state"/>
        <label ref="state"/>
      </itemset>
  </select1>
  </h:body>
</h:html>

CSV (cities.csv)

name,label,state,county
dumont,Dumont,texas,king
finney,Finney,texas,king
brownsville,brownsville,texas,cameron
harlingen,harlingen,texas,cameron
seattle,Seattle,washington,king
redmond,Redmond,washington,king
tacoma,Tacoma,washington,pierce
puyallup,Puyallup,washington,pierce

ODK Aggregate throws the above validation error when I try to upload this XForm. The XLS results in the below validation error when trying to convert to XML using pyxform 0.9.23.

Traceback (most recent call last):
  File "xls2xform.py", line 94, in <module>
    main_cli()
  File "xls2xform.py", line 86, in main_cli
    args.path_to_XLSForm, args.output_path, args.skip_validate)
  File "xls2xform.py", line 23, in xls2xform_convert
    survey.print_xform_to_file(xform_path, validate=validate, warnings=warnings)
  File "/src/pyxform/pyxform/survey.py", line 507, in print_xform_to_file
    warnings.extend(check_xform(path))
  File "/src/pyxform/pyxform/odk_validate/__init__.py", line 133, in check_xform
    'ODK Validate Errors:\n' + _cleanup_errors(stderr))
odk_validate.ODKValidateError: ODK Validate Errors:
org.javarosa.xform.parse.XFormParseException: <label> node for itemset doesn't exist! [instance(cities)/root/item/itextId]

>> XForm is invalid. See above for the errors.

Result: Invalid

[MartijnR]

Thanks @sjudeng,

Did pyxform produce this XForm output? If so, pyxform has a bug.

This part should be corrected to:

        <value ref="name"/>
        <label ref="label"/>

live: https://enke.to/::YYU9

[sjudeng]

No, pyxform didn't produce it ... I was hand generating the XForm and I misunderstood what goes into those fields. Thanks for this fix, it works now!

Is it possible to do this through XLSForm, or does it currently require custom XML? I get the above validation error (e.g. <label> node for itemset doesn't exist! [instance(cities)/root/item/itextId]) when I try to convert this XLSForm.

| survey |                                 |       |       |
|        | type                            | name  | label |
|        | select_one_from_file cities.csv | state | state |

[MartijnR]

It should currently not require custom XML anymore in the latest pyxform (but nobody is using this yet, not announced, so you're a pioneer). There doesn't seem to be anything wrong with your XLSForm so there may be a bug in pyxform after all. If possibly, try running it with ODK Validate disabled (I think this is possible - not sure) and see the XForm output. (I'm getting ready to go on a holiday, so I won't be able to spend much time on this myself until I'm back).

[sjudeng]

Below is the pyxform output with --skip_validate. Thanks again for your quick responses on this. Getting this working, even if just with XML for now, is great. Enjoy your holiday.

<?xml version="1.0"?>
<h:html xmlns="http://www.w3.org/2002/xforms" xmlns:ev="http://www.w3.org/2001/xml-events" xmlns:h="http://www.w3.org/1999/xhtml" xmlns:jr="http://openrosa.org/javarosa" xmlns:orx="http://openrosa.org/xforms" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <h:head>
    <h:title>select from file test</h:title>
    <model>
      <instance>
        <select_from_file_test id="select_from_file_test">
          <state/>
          <meta>
            <instanceID/>
          </meta>
        </select_from_file_test>
      </instance>
      <instance id="cities" src="jr://file-csv/cities.csv">
        <root>
          <item>
            <name/>
            <label/>
          </item>
        </root>
      </instance>
      <bind nodeset="/select_from_file_test/state" type="select1"/>
      <bind calculate="concat('uuid:', uuid())" nodeset="/select_from_file_test/meta/instanceID" readonly="true()" type="string"/>
    </model>
  </h:head>
  <h:body>
    <select1 ref="/select_from_file_test/state">
      <label>state</label>
      <itemset nodeset="instance('cities')/root/item">
        <value ref="name"/>
        <label ref="jr:itext(itextId)"/>
      </itemset>
    </select1>
  </h:body>
</h:html>

[MartijnR]

Great, thanks! It seems the issue is that pyxform creates a non-existing itext reference instead of directly referencing the node. That's definitely a bug.