Add support for external instances #30

MartijnR · 2015-02-27T23:16:03Z

Support for both select_one_from_file [FILENAME] and select_multiple_from_file [FILENAME]
The above types always create regular <select> or <select1> with a regular <itemset> and always add a single <instance id="" src="" /> to the model (unless that instance with that id already exists).
Filenames can have either .csv or .xml extensions.
instance id attribute = lowercased filename minus the extension: countries.csv => <instance id="countries">
instance src attribute for XML sources gets jr://file/ prefix and for CSV sources gets jr://file-csv/ prefix (similar to jr://image, jr://video and jr://audio).
the label ref of an itemset is always ref="label" because itext references couldn't work.

Examples

1a. XLSForm input (CSV file, no choice filter):

type	name	label	choice_filter
select_one_from_file countries.csv	country	Pick one country

1b. XForm output:

Very similar to a regular select_one with choice filter (but in this case the choice filter is empty)

<model>
    <instance>
        ....
    </instance>
    <instance id="cities" src="jr://file-csv/cities.csv" />
</model>
<h:body>
    <select1 ref="/myform/country">
        <label>Country</label>
        <itemset nodeset="instance('countries')/root/item">
            <value ref="name"/>
            <label ref="label"/>
        </itemset>
    </select1>
</h:body>

2a. XLSForm input (XML file, no choice filter):

type	name	label	choice_filter
select_one_from_file countries.xml	country	Pick one country

2b. XForm output:

Exactly the same syntax as used by CommCare for their many XML external instances

<model>
    <instance>
        ....
    </instance>
    <instance id="cities" src="jr://file/cities.xml" />
</model>
<h:body>
    <select1 ref="/myform/country">
        <label>Country</label>
        <itemset nodeset="instance('countries')/root/item">
            <value ref="name"/>
            <label ref="label"/>
        </itemset>
    </select1>
</h:body>

3a. XLSForm input (CSV file, with choice filter):

type	name	label	choice_filter
select_multiple_from_file cities.csv	city	Pick some cities	country = ${country}

3b. XForm output:

<model>
    <instance>
        ....
    </instance>
    <instance id="cities" src="jr://file-csv/cities.csv" />
</model>
<h:body>
    <select ref="/myform/country">
        <label>Country</label>
        <itemset nodeset="instance('cities')/root/item[country= /myform/country ]">
            <value ref="name"/>
            <label ref="label"/>
        </itemset>
    </select>
</h:body>

Notes:

this feature is not compatible with ODK Collect (and vice versa select_external and search are not compatible with Enketo)
the external data referred to in an XLSForm needs to always have a label column (that's an XLSForm restriction)
csv column headers may not contain a colon (this not relevant to Pyxform but to the xlsform.org documentation)
these XForms do not pass ODK Validate validation (I suspect it checks whether itemset nodeset paths exist), so to use this feature ODK Validate would have to be switched off or tweaked so it bypasses checks for paths that refer to external instances (i.e. that refer to instance(ID)/path/to/node when the targeted instance has a src attribute)

The text was updated successfully, but these errors were encountered:

ukanga · 2015-03-01T21:23:56Z

Any chance this will be added to javarosa/odk validate/odk collect family?

MartijnR · 2015-03-01T22:58:23Z

Hopefully. Otherwise we need to go ahead with building a new validator based around's enketo's XPath evaluator. I know Dimagi is using external XML instances a lot already (same as above). Not sure what validator they use.

The feature has now been added to Enketo Express.

mberg · 2015-03-02T06:38:49Z

Martijn are you following the approach dimagi is using? Do they have their
own version of javarosa to support this?
On Mar 2, 2015 1:58 AM, "Martijn van de Rijdt" [email protected]
wrote:

Hopefully. Otherwise we need to go ahead with building a new validator
based around's enketo's XPath evaluator
enketo/enketo-validate#1. I know Dimagi is
using external XML instances a lot already (same as above). Not sure what
validator they use.

The feature is has now been added to Enketo Express.

—
Reply to this email directly or view it on GitHub
#30 (comment).

MartijnR · 2015-03-02T17:47:27Z

Yes, same approach. No re-inventing of any wheels. I just added CSV support in addition to XML. Dimagi has their own version of Javarosa that supports XML external instances as above.

dorey · 2015-03-18T19:59:09Z

👍 for referencing the URI of the external itemset from within the XForm.

MartijnR · 2015-03-18T20:26:03Z

This opens the way for some exciting future possibilities, because the URL published in the /xFormsManifest could be a dynamic document. E.g. a dynamically created XML 👍 (or CSV 👎 ) Document.... It could be external to KoBo/Ona too.

You could e.g. create a service that dynamically creates a document based on another survey or based on earlier responses to the same survey.

mberg · 2015-03-19T06:36:01Z

Is this working?

On Wednesday, March 18, 2015, Martijn van de Rijdt [email protected]
wrote:

This opens the way for some exciting future possibilities, because the URL
posted in the /xFormsManifest could be a dynamic document. E.g. a
dynamically created XML (or CSV [image: 👎] ) Document.... It could be
external to KoBo/Ona too.

You could e.g. create a service that dynamically creates a document based
on another survey or earlier responses to the same survey.

—
Reply to this email directly or view it on GitHub
#30 (comment).

MartijnR · 2015-03-19T14:31:52Z

In Enketo Express it is working, yes.

mberg · 2015-03-19T15:20:38Z

Awesome

On Thursday, March 19, 2015, Martijn van de Rijdt [email protected]
wrote:

In Enketo Express it is working, yes.

—
Reply to this email directly or view it on GitHub
#30 (comment).

MartijnR · 2015-04-09T22:14:07Z

One way around ODK Validate is to add dummy content to the external instance. Instead of:

<instance id="cities" src="jr://file/cities.xml" />

we do:

<instance id="cities" src="jr://file/cities.xml" >
    <!-- the following line is necessary for some xform parsers -->
    <root><item><name/><label/></item></root>
</instance>

It's just a temporary dirty trick to fool ODK Validate, but it should work for all Pyxform-produced forms. Luckily the javarosa parser doesn't check the content of the choice_filter.

…ribute, XLSForm/pyxform#30

MartijnR · 2015-06-15T15:17:57Z

Bump @dorey, @ukanga, @tinok, @mberg. Who is going to add this to pyxform?

Currently this Enketo feature is only supported by ODK Aggregate.

mberg · 2015-06-15T15:46:25Z

What's needed? Is there a ticket for this?

On Monday, June 15, 2015, Martijn van de Rijdt [email protected]
wrote:

Bump @dorey https://github.com/dorey, @ukanga
https://github.com/ukanga, @tinok https://github.com/tinok, @mberg
https://github.com/mberg. Who is going to add this to pyxform?

Currently this Enketo feature is only supported by ODK Aggregate.

—
Reply to this email directly or view it on GitHub
#30 (comment).

dorey · 2015-06-15T18:05:04Z

I wrote it out into a test here using the 'pyxform_test_case' syntax ( from pr #28 ).

@MartijnR, If that test describes all that's required, we can take a stab at it in the next couple weeks.

MartijnR · 2015-06-15T19:25:52Z

beautiful @dorey 👍 The only thing to change in the tests, is the ODK Validate trick, either by removing the self-closing / in the model_contains tests or by adding the <root><item><name/><label/></item></root> template. Former might be more attractive perhaps.

mberg · 2015-07-02T14:55:00Z

Martijn,

Would this enable pull csv like functionality in enketo express?

Matt

On Mon, Jun 15, 2015 at 10:25 PM, Martijn van de Rijdt <
[email protected]> wrote:

beautiful @dorey https://github.com/dorey [image: 👍] The only thing
to change in the tests, is the ODK Validate trick, either by removing the
self-closing / in the model_contains tests or by adding the
template. Former might be more
attractive perhaps.

—
Reply to this email directly or view it on GitHub
#30 (comment).

MartijnR · 2015-07-02T15:44:36Z

[removed earlier reply]

pulldata() support has now been added to both Enketo (and pyxform) by adding the external <instance> with a src attribute.

MartijnR · 2016-01-04T19:04:06Z

UPDATED: ignore this comment

I wonder if, after the recent changes wrt to column 'value' we should output instead:

<value ref="value"/> <!-- ref="value" instead of "name" -->
<label ref="label"/>

This would change the ODK-validate trick into:

<instance id="cities" src="jr://file/cities.xml" >
    <!-- the following line is necessary for some xform parsers -->
    <root><item><value/><label/></item></root>
</instance>

and thus require a value and label column (csv) or node (XML) in the external document.

MartijnR · 2016-08-03T21:22:43Z

Actually "value" instead of "name" doesn't seem to work in PyxForm (for regular choices sheet). So let's stick with using "name". In any case, any OpenRosa XForm client supporting this feature could deal with both as it's just regular XForm syntax. So it could quite easily be changed later, without breaking existing forms.

The snippets in the OP are correct.

Sample XForm + XML media: external_xml.zip and live
Sample XForm + CSV media: external_csv.zip and live

ukanga · 2017-04-04T15:59:26Z

@MartijnR I notice that the form.xml in external_xml.zip makes use of jr://file/cities.xml and what pyxform currently generates is jr://file-xml/cities.xml. Is this going to be an issue?

Also, what content type does enketo expect? The XML example https://enketo-stage.ona.io/::YYmJ seems Enketo is attempting a CSV processing instead of XML.

MartijnR · 2017-04-04T16:44:46Z

@ukanga, when looking at the code only, it seems like Enketo may (accidentally) be lenient towards this. However, jr:file-xml is definitely not correct. Sorry I missed this. For XML files it should be jr://file/cities.xml. (PS. this has now become part of the official ODK XForms spec)

Content-type should be appropriate for the source. At the moment, in Enketo if the (jquery get) auto-parsed result is not an XML Document it will be considered CSV. I think generally it should be text/xml or application/xml for XML and text/csv for CSV.

MartijnR · 2017-04-04T16:48:54Z

I've added an issue to EE as the explicitness of the jr:file or jr:file-csv connectors should probably be used to parse correctly regardless of the Content-Type it is served with. kobotoolbox/enketo-express#710

ukanga · 2017-04-04T18:39:00Z

Thanks @MartijnR. I hope this is appropriate then #106.

MartijnR added a commit to enketo/enketo-core that referenced this issue Apr 9, 2015

fixed: remove any existing content from instances that have a src att…

db1fd1d

…ribute, XLSForm/pyxform#30

MartijnR mentioned this issue Jul 8, 2015

Does Enketo Express support the pulldata function? kobotoolbox/enketo-express#250

Closed

MartijnR closed this as completed Jan 4, 2016

MartijnR reopened this Jan 4, 2016

ukanga added a commit that referenced this issue Aug 7, 2016

add support for external instances, closes #30

fab7fce

ukanga mentioned this issue Aug 7, 2016

add support for external_instances #76

Merged

dorey closed this as completed in #76 Aug 11, 2016

sjudeng mentioned this issue Aug 26, 2016

External instance filtering with Enketo and ODK Aggregate kobotoolbox/enketo-express#545

Closed

lognaturel mentioned this issue Feb 3, 2017

Proposal: External secondary instances (as in CommCare) getodk/xforms-spec#50

Closed

rhunwicks mentioned this issue Apr 3, 2017

Attachments using a media URL instead of an attached file should still have a filename kobotoolbox/kobocat#344

Closed

lognaturel mentioned this issue Apr 10, 2019

Document select_one_from_file / select_multiple_from_file XLSForm/xlsform.github.io#164

Closed

jnm mentioned this issue Apr 25, 2019

Problems XLSForms referencing external .CSV files kobotoolbox/kobo-docker#183

Closed

shankari mentioned this issue May 22, 2019

Restore Survey Answers e-mission/e-mission-docs#404

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for external instances #30

Add support for external instances #30

MartijnR commented Feb 27, 2015 •

edited

Loading

ukanga commented Mar 1, 2015

MartijnR commented Mar 1, 2015

mberg commented Mar 2, 2015

MartijnR commented Mar 2, 2015

dorey commented Mar 18, 2015

MartijnR commented Mar 18, 2015 •

edited

Loading

mberg commented Mar 19, 2015

MartijnR commented Mar 19, 2015

mberg commented Mar 19, 2015

MartijnR commented Apr 9, 2015

MartijnR commented Jun 15, 2015

mberg commented Jun 15, 2015

dorey commented Jun 15, 2015

MartijnR commented Jun 15, 2015

mberg commented Jul 2, 2015

MartijnR commented Jul 2, 2015

MartijnR commented Jan 4, 2016 •

edited

Loading

MartijnR commented Aug 3, 2016 •

edited

Loading

ukanga commented Apr 4, 2017 •

edited

Loading

MartijnR commented Apr 4, 2017 •

edited

Loading

MartijnR commented Apr 4, 2017 •

edited

Loading

ukanga commented Apr 4, 2017

Add support for external instances #30

Add support for external instances #30

Comments

MartijnR commented Feb 27, 2015 • edited Loading

Examples

1a. XLSForm input (CSV file, no choice filter):

1b. XForm output:

2a. XLSForm input (XML file, no choice filter):

2b. XForm output:

3a. XLSForm input (CSV file, with choice filter):

3b. XForm output:

ukanga commented Mar 1, 2015

MartijnR commented Mar 1, 2015

mberg commented Mar 2, 2015

MartijnR commented Mar 2, 2015

dorey commented Mar 18, 2015

MartijnR commented Mar 18, 2015 • edited Loading

mberg commented Mar 19, 2015

MartijnR commented Mar 19, 2015

mberg commented Mar 19, 2015

MartijnR commented Apr 9, 2015

MartijnR commented Jun 15, 2015

mberg commented Jun 15, 2015

dorey commented Jun 15, 2015

MartijnR commented Jun 15, 2015

mberg commented Jul 2, 2015

MartijnR commented Jul 2, 2015

MartijnR commented Jan 4, 2016 • edited Loading

MartijnR commented Aug 3, 2016 • edited Loading

ukanga commented Apr 4, 2017 • edited Loading

MartijnR commented Apr 4, 2017 • edited Loading

MartijnR commented Apr 4, 2017 • edited Loading

ukanga commented Apr 4, 2017

MartijnR commented Feb 27, 2015 •

edited

Loading

MartijnR commented Mar 18, 2015 •

edited

Loading

MartijnR commented Jan 4, 2016 •

edited

Loading

MartijnR commented Aug 3, 2016 •

edited

Loading

ukanga commented Apr 4, 2017 •

edited

Loading

MartijnR commented Apr 4, 2017 •

edited

Loading

MartijnR commented Apr 4, 2017 •

edited

Loading