Adding tests for BigML pmml models

[mmerce]

Adding test code for some PMML models generated by BigML. We only managed to run the tests sucessfully when the headers row in the CSV files contains the names of the fields, but it would be great if we could also use the "displayName" of the DataDictionary instead. Any comment on how to do that would be greatly appreciated.

Thanks!

[vruusmann]

Do you have a script (WhizzML?), which would automate the model training and PMML export using BigML APIs? It would be highly desirable to automate this process. For example, when I detect that BigML has updated their service versions, then I could re-run this script, and immediately get updated CSV and PMML files?

This would allow me/us to make a claim that "JPMML-Evaluator has been tested with the latest BigML version".

We only managed to run the tests sucessfully when the headers row in the CSV files contains the names of the fields

Unfortunately, this is how JPMML-Evaluator integration tests are set up - CSV columns are mapped to PMML fields based on their name attribute.

it would be great if we could also use the "displayName" of the DataDictionary instead.

Here's the definition of the target field from your DecisionTreeIris.pmml file:

<DataField dataType="string" displayName="species" name="000004" optype="categorical">
    <Value value="Iris-setosa"/>
    <Value value="Iris-versicolor"/>
    <Value value="Iris-virginica"/>
</DataField>

Most PMML producer software do not use the DataField@displayName attribute at all, and try to keep the value of the DataField@name attribute sufficiently human-readable. For example, I would simplify the above field definition to this:

<DataField dataType="string" name="species" optype="categorical">
    <Value value="Iris-setosa"/>
    <Value value="Iris-versicolor"/>
    <Value value="Iris-virginica"/>
</DataField>

Any chance that BigML would be adopting this convention? Of course, it's always possible to modify the PMML file (updating DataField element names, and all their references) using Visitor APIs from the JPMML-Model library.

[mmerce]

Hi Villu, thanks for your quick answer!

Do you have a script (WhizzML?), which would automate the model training and PMML export using BigML APIs? It would be highly desirable to automate this process. For example, when I detect that BigML has updated their service versions, then I could re-run this script, and immediately get updated CSV and PMML files?

We can indeed automate the generation of any resource starting from remotely publicly accessible CSVs. However, that would require anyone who wanted to test to be registered in BigML and use their credentials when connecting to BigML to do all the process. Also, WhizzML scripts are shareable but need to be owned to be run, so you would need the script to be created or cloned first in your account. I think it's better to start simple and add the pmml output as I did in the PR. I'm the main maintainer of BigML's bindings, and I was planning on updating the repo with any major change we add.

Most PMML producer software do not use the DataField@displayName attribute at all, and try to keep the value of the DataField@name attribute sufficiently human-readable.

Yes, but in BigML you can use any data file, even if it has duplicated column names. That's why we associate an ID to each column, and this is the text used as name in our DataField. Of course, we can create a customized output specially for jpmml-evaluator, but as this restricts a bit the abilities of our product we hoped that by using the displayName we could maintain our full flexibility. Do I get it right and it's only because of an assumption made in the test code? Can the displayName be used in production code?

Thanks again!

[vruusmann]

That's why we associate an ID to each column, and this is the text used as name in our DataField.

The ID is BigML-specific metadata, and could/should better be stored for future reference using the Extension element:

<DataField dataType="string" name="species" optype="categorical">
    <Extension name="BigML-COLUMN_ID" value="000004"/>
    <Value value="Iris-setosa"/>
    <Value value="Iris-versicolor"/>
    <Value value="Iris-virginica"/>
</DataField>

You may choose any other extension name (instead of BigML-COLUMN_ID). Just document it somewhere, and keep using it consistently.

Of course, we can create a customized output specially for jpmml-evaluator

I would argue that every PMML consumer out there (see http://dmg.org/pmml/products.html) expects that CSV column names can be directly mapped to DataField@name attribute values.

The DataField@displayName is more like a human-readable description, not a unique identifier. Most of the time the DataField@displayName attribute is not defined altogether.

Do I get it right and it's only because of an assumption made in the test code?

BigML is currently more in the role of PMML producer. It would be a very smart move to produce PMML markup that is supported/recognizable by the majority of PMML consumers already in existence (see my above link/comment).

[mmerce]

Thanks for the suggestion. We'll work on it and add the changes to this PR when ready.

[mmerce]

The PMML output has been adapted to the expected syntax for fields and some more models have been included.