xlsform / pyxform

A Python package to create XForms for ODK Collect.

pyxform

pyxform is a Python library that simplifies writing forms for ODK Collect and Enketo by converting spreadsheets that follow the XLSForm standard into ODK XForms. The XLSForms format is used in a number of tools.

Project status

pyxform is actively maintained by ODK.

Current goals for the project include:

• Enable more complex workflows through sophisticated XPath expressions and entities
• Improve error messages and make troubleshooting easier
• Improve experience, particularly for multi-language forms

pyxform was started at the Sustainable Engineering Lab at Columbia University, and until 2018 was maintained primarily by Ona.

Using pyxform

For user support, please start by posting to the ODK forum where your question will get the most visibility.

There are 3 main ways that pyxform is used:

• Through a form server, such as the pyxform-http service wrapper, or ODK Central.
• The command line utility xls2xform, which can be helpful for troubleshooting or as part of a broader form creation pipeline.
• As a library, meaning that another python project imports functionality from pyxform.

Running the latest release of pyxform

To convert forms at the command line, the latest official release of pyxform can be installed using pip:

pip install pyxform

The xls2xform command can then be used:

xls2xform path_to_XLSForm [output_path]

The currently supported Python versions for pyxform are 3.10, 3.11 and 3.12.

Running pyxform from local source

Note that you must uninstall any globally installed pyxform instance in order to use local modules. Please install java 8 or newer version.

From the command line, complete the following. These steps use a virtualenv to make dependency management easier, and to keep the global site-packages directory clean:

# Get a copy of the repository.
mkdir -P ~/repos/pyxform
cd ~/repos/pyxform
git clone https://github.com/XLSForm/pyxform.git repo

# Create and activate a virtual environment for the install.
/usr/local/bin/python3.10 -m venv venv
. venv/bin/activate

# Install the pyxform and it's production dependencies.
(venv)$ cd repo
# If this doesn't work, upgrade pip ``pip install --upgrade pip`` and retry.
(venv)$ pip install -e .
(venv)$ python pyxform/xls2xform.py --help
(venv)$ xls2xform --help           # same effect as previous line
(venv)$ which xls2xform            # ~/repos/pyxform/venv/bin/xls2xform

To leave and return to the virtualenv:

(venv)$ deactivate                 # leave the venv, scripts not on $PATH
$ xls2xform --help
# -bash: xls2xform: command not found
$ . ~/repos/pyxform/venv/bin/activate     # reactivate the venv
(venv)$ which xls2xform                   # scripts available on $PATH again
~/repos/pyxform/venv/bin/xls2xform

Installing pyxform from remote source

pip can install from the GitHub repository. Only do this if you want to install from the master branch, which is likely to have pre-release code. To install the latest release, see above.:

pip install git+https://github.com/XLSForm/pyxform.git@master#egg=pyxform

You can then run xls2xform from the commandline:

xls2xform path_to_XLSForm [output_path]

Development

To set up for development / contributing, first complete the above steps for "Running pyxform from local source". Then repeat the command used to install pyxform, but with [dev] appended to the end, e.g.:

pip install -e .[dev]

You can run tests with:

python -m unittest

Before committing, make sure to format and lint the code using ruff:

ruff format pyxform tests
ruff check pyxform tests

If you are using a copy of ruff outside your virtualenv, make sure it is the same version as listed in pyproject.toml. Use the project configuration for ruff in pyproject.toml, which occurs automatically if ruff is run from the project root (where pyproject.toml is).

Contributions

We welcome contributions that have a clearly-stated goal and are tightly focused. In general, successful contributions will first be discussed on the ODK forum or in an issue. We prefer discussion threads on the ODK forum because pyxform issues generally involve considerations for other tools and specifications in ODK and its broader ecosystem. Opening up an issue or a pull request directly may be appropriate if there is a clear bug or an issue that only affects pyxform developers.

Writing tests

Make sure to include tests for the changes you're working on. When writing new tests you should add them in tests folder. Add to an existing test module, or create a new test module. Test modules are named after the corresponding source file, or if the tests concern many files then module name is the topic or feature under test.

When creating new test cases, where possible use PyxformTestCase as a base class instead of unittest.TestCase. The PyxformTestCase is a toolkit for writing XLSForms as MarkDown tables, compiling example XLSForms, and making assertions on the resulting XForm. This makes code review much easier by putting the XLSForm content inline with the test, instead of in a separate file. A unittest.TestCase may be used if the new tests do not involve compiling an XLSForm (but most will). Do not add new tests using the old style XFormTestCase.

When writing new PyxformTestCase tests that make content assertions, it is strongly recommended that the xml__xpath* matchers are used, in particular xml__xpath_match. Most older tests use matchers like xml__contains and xml__excludes, which are simple string matches of XML snippets against the result XForm. The xml__xpath_match kwarg accepts an XPath expression and expects 1 match. The main benefits of using XPath are 1) it allows specifying a document location, and 2) it does not require a particular document order for elements or attributes or whitespace output. To take full advantage of 1), the XPath expressions should specify the full document path (e.g. /h:html/h:head/x:model) rather than a search (e.g. .//x:model). To take full advantage of 2), the expression should include element predicates that specify the expected attribute values, e.g. /h:html/h:body/x:input[@ref='/trigger-column/a']. To specify the absence of an element, an expression like the following may be used with xml__xpath_match: /h:html[not(descendant::x:input)], or alternatively xml__xpath_count: .//x:input with an expected count of 0 (zero).

Documentation

For developers, pyxform uses docstrings, type annotations, and test cases. Most modern IDEs can display docstrings and type annotations in a easily navigable format, so no additional docs are compiled (e.g. sphinx). In addition to the user documentation, developers should be familiar with the ODK XForms Specification https://getodk.github.io/xforms-spec/.

For users, pyxform has documentation at the following locations: * XLSForm docs * XLSForm template * ODK Docs

Change Log

Changelog

Releasing pyxform

1. Make sure the version of ODK Validate in the repo is up-to-date:

   pyxform_validator_update odk update ODK-Validate-vx.x.x.jar
   

2. Run all tests through Validate by setting the default for run_odk_validate to kwargs.get("run_odk_validate", True) in tests/pyxform_test_case.py.

3. Draft a new GitHub release with the list of merged PRs. Follow the title and description pattern of the previous release.

4. Checkout a release branch from latest upstream master.

5. Update CHANGES.txt with the text of the draft release.

6. Update pyproject.toml, pyxform/__init__.py with the new release version number.

7. Commit, push the branch, and initiate a pull request. Wait for tests to pass, then merge the PR.

8. Tag the release and it will automatically be published

Manually releasing

Releases are now automatic. These instructions are provided for forks or for a future change in process.

1. In a clean new release only directory, check out master.

2. Create a new virtualenv in this directory to ensure a clean Python environment:

   /usr/local/bin/python3.10 -m venv pyxform-release
   . pyxform-release/bin/activate
   

3. Install the production and packaging requirements:

   pip install -e .
   pip install flit==3.9.0
   

4. Clean up build and dist folders:

   rm -rf build dist pyxform.egg-info
   

5. Prepare sdist and bdist_wheel distributions, and publish to PyPI:

   flit --debug publish --no-use-vcs
   

6. Tag the GitHub release and publish it.

Related projects

These projects are not vetted or endorsed but are linked here for reference.

Converters

To XLSForm

• cueform (Go): from CUE
• md2xlsform (Python): from MarkDown
• xlsform (Python): from JSON
• yxf (Python): from YAML

From XLSForm

• ODK2Doc (R): to Word
• OdkGraph (Python): to a graph
• Pureser (Swift): to HTML
• ppp (Python): to HTML, PDF, Word
• QuestionnaireHTML (R): to HTML
• xlsform-converter (Python): to Django modules
• xlsform (Python): to JSON
• xlsform2json (Java): to JSON
• XLSform2PDF (Python): to PDF
• xlson (Python): to OpenSRP JSON
• yxf (Python): to YAML

Management Tools

• surveydesignr (R): compare XLSForms
• ipacheckscto (Stata): check XLSForm for errors or design issues
• kobocruncher (R): generate analysis Rmd from XLSForm
• odkmeta (Stata): use XLSForm to import ODK data to Stata
• odktools (C++): convert pyxform internal data model to MySQL
• pmix (Python): manage XLSForm authoring
• pyxform-docker (Dockerfile): image for pyxform development
• xform-test (Java): test XLSForms
• xlsformpo (Python): use .po files for XLSForm translations
• XlsFormUtil (R): manage XLSForm authoring

pyxform's Issues

PYXForm should fail if a choiceList has two or more choices sharing the same value.

When processing the XLSX file, it would be helpful if the conversion failed if a user has entered two rows in the choices sheet for a given choice list that have the same value.

The use case is someone copy-and-pasting a series of answers:

listA val1 Yes
listA val2 No
listA val2 Maybe

and not noticing that val2 is repeated twice (Maybe should be val3).

If we want to support the current mechanism where this is allowed, we should probably have
a settings tab flag that allows for duplicated choice values.

Default language for labels and constraint messages when explicit default language is not specified

Given a xlsform where the labels and constraint messages are present in multiple languages and there is no explicit default language specified.
My question is:
In the generated xform what would be the default/first language?

allow "value" as an alias for "name" in survey sheet

see discussion in odk-developers group

Add support for quotes

"For my scenario, I would like to allow single quote (') in the names, for names such as O'Brian.

When I try to convert that, I get an error and I think it's because of the single quote in the middle. I tried using double quotes but kept getting the same error."

appearance=table-list breaks if additional appearance is added

Appearance table-list is properly detected, resulting correctly in inputs with appearance="label" and appearance="list-nolabel"

However, when adding another appearance, e.g. table-list something, it appears to not be detected. The output does not add appearance="label" and appearance="list-nolabel" to inputs in this group.

test form

Background: just something I happened to see when testing the boundaries of a new 'random' feature for select and select1 questions. 'appearance="random" is not actually planned to be supported in tables at least not initially ever.

Update:

a randomize() XPath function was introduced instead which is much better. This issue still stands though

2 translated media labels outputs a 3rd (default) language

Form source here: https://docs.google.com/spreadsheets/d/1HlxpC4ORzor94WXONUqJXbrrlrsSQAEXdGsiAviM4lU/edit?usp=sharing

The default language has no valid audio src (because it doesn't exist in the XLSForm).

XForm itext output is:

<itext>
    <translation default="true()" lang="default">
        .....
    </translation>
    <translation lang="Russian">
        ......      
    </translation>
    <translation lang="Kyrgyz">
       .......
    </translation>
</itext>

But expected itext output is:

<itext>
    <translation lang="Russian">
        .....
    </translation>
    <translation lang="Kyrgyz">
       .......
    </translation>
</itext>

Discussion: 'value' instead of 'name' in the choices sheet

Leave 'name' as alias for 'value' but change documentation to 'value'.

I think this is more intuitive for end users.

What do you think?

Hint text for repeats is ignored

Hint text specified for a begin repeat is ignored.

A <hint> element within the group created for the repeat should be created.

e.g.

would create

    <group ref="/Families/pets">
      <label>Pets</label>
      <hint>Pet details</hint>
      <repeat nodeset="/Families/pets">
        <input ref="/Families/pets/pets_name">
          <label>Pet's Name</label>
          <hint>Pets name hint</hint>
        </input>
        <select1 ref="/Families/pets/pet_type">
          <label>Type of pet</label>
          <hint>Type of pet hint</hint>
          <item>
            <label>Dog</label>
            <value>dog</value>
          </item>
          <item>
            <label>Cat</label>
            <value>cat</value>
          </item>
          <item>
            <label>Bird</label>
            <value>bird</value>
          </item>
          <item>
            <label>Fish</label>
            <value>fish</value>
          </item>
        </select1>
        <upload mediatype="image/*" ref="/Families/pets/pet_picture">
          <label>Picture of pet</label>
          <hint>Take a nice photo</hint>
        </upload>
      </repeat>
    </group>

Non-ascii characters in a question name cause an exception

When non-ascii characters are in the question name, the conversion fails with an exception such as -

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 1: ordinal not in range(128)

instead of reporting that the question name is invalid.

Deeply nested groups effectively freeze pyxform

If you have groups nested 5-10 levels deep, pyxform begins to take hours to process the form (on the fastest of systems). It appears that pyxform has frozen completely, but it will eventually complete.

The fix is in section.py. Change this:

    return [e.xml_control() for e in self.children if e.xml_control() is not None]

To this:

    for e in self.children:
        control = e.xml_control()
        if control is not None:
            yield control

Add support for custom settings

Similar to how #65 discusses the ability to add custom namespaces to support custom bind:: and body:: properties for questions, it would be very useful to be able to add custom global properties in the settings sheet.

e.g. The column head::esri::somethingglobal in settings containing the value The Value might translate into the xml as

<?xml version="1.0"?>
<h:html xmlns="http://www.w3.org/2002/xforms" xmlns:esri="http://esri.com/xforms" xmlns:ev="http://www.w3.org/2001/xml-events" xmlns:h="http://www.w3.org/1999/xhtml" xmlns:jr="http://openrosa.org/javarosa" xmlns:orx="http://openrosa.org/xforms/" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <h:head>
    <h:title>My Survey</h:title>
    <esri:somethingglobal>The value</esri:somethingglobal>

   ...

non-existing itext reference

Using select_one_from_file creates an itext reference for the label (under certain conditions?). It should directly reference the node name instead.

This XLSForm produces:

<itemset nodeset="instance('cities')/root/item">
    <value ref="name"/>
    <label ref="jr:itext(itextId)"/>
</itemset>

but should produce (never an itext reference):

<itemset nodeset="instance('cities')/root/item">
    <value ref="name"/>
    <label ref="label"/>
</itemset>

Thanks for finding this @sjudeng!

Support for numeric range questions

Add support for numeric range questions.

The XForms specification describes the <range> control https://en.wikibooks.org/wiki/XForms/Range which might look like

This would be provide more flexibility than the distress appearance as supported in Enketo which has a limited range of 0 to 10.

The <range> control is also mentioned in the JavaRosa roadmap https://bitbucket.org/javarosa/javarosa/wiki/xform

Two approaches could be considered -

Option 1

A range question type, which would be supported by additional columns to define the parameters for start, end, step and increment and appearance type to control presentation (eg compact for spinner control or horizontal for a horizontal bar)

Option 2

Support a range appearance type for integer and decimal question types.

e.g range(start, end, step, increment) with logical default for all parameters (e.g. 1, 10, 1, 1)

Option 2 has some simplicity advantages in that does not require additional columns to be recognised nor the addition of new control types. It could also be combined with other appearance for additional styling control. As its just an appearance type, there would no change to pyxform.

e.g. range(1, 200) compact

Which might look like -

An enhanced range(...) appearance bears some similarity in concept to the select(...) expression appearance type http://xlsform.org/#dynamic-selects-from-pre-loaded-data

Support PDF File Upload question type

Clear error message when processing a file with spaces in the filename e.g 'tutorial .xls'

currently it states invalid xml tag.

Add license

I think this repository needs a license.

big-image and media::big-image column heading for itext tag is not available in XLSForm

There is a "big-image" itext value tag type that is not supported by XLSForm.

Usage section below is copied from this page: https://opendatakit.org/help/form-design/guidelines/

---

Itext value tag types

The possible choices for the form attribute of the itext tag are:

"long",
"short",
"image",
"audio",
"video", and
"big-image".

By default, itext "image" values are not clickable. However, if you also include a "big-image", the image displayed by "image" will be clickable and will display a pannable, zoomable view of the file specified by "big-image". You can return to your form after opening a "big-image" by hitting the phone's "back" button. Specifying "big-image" alone has no effect, you must always include "image".

Files referenced by "image" and "big-image" may be the same; however, to improve the user experience, we highly recommend creating smaller thumbnail images to be referenced by "image" (images will display faster if they do not need to be dramatically scaled down).

Add external instance when pulldata function is used

This will create a reference to the external data in the XForm which means that strict XForm clients (such as Enketo) will be able to use this data. This means those clients could e.g. start supporting the pulldata() function without data being available magically.

Search for pulldata expressions in the calculation column, e.g. pulldata('hhplotdata', 'plot1size', 'hhid_key', ${hhid}).

If found:
0. extract id (first parameter of pulldata expression), in the example: "hhplotdata"

1. check if <instance id="hhplotdata"> node exists
2. if not, add secondary instance to XForm output:

<instance id="hhplotdata" src="jr://file-csv/hhplotdata.csv" />

version bump 0.9.24

• fix non-existent itext refence #80

Provide feedback when invalid public_key is used

If you build a form with an invalid key, it can be a major pain to debug why surveys aren't being encrypted.

Support select one or_other when we have a choice_filter

Add support for line breaks

I have a client who puts line breaks into calculate statements in excel as but xlsform converts the ampersand to &

"We mainly use this for display the labels of the selected choices of a select multiple in the label of a later question so that each appears on it’s own line. For each selected option, I concatenate jr:choice-name with these line breaks for each possible choice."

Support XForms OSM Upload Media Type

• documentation: https://github.com/AmericanRedCross/openmapkit/wiki/XForms-OSM-Upload-Media-Type
• example xlsform

do not trim newline characters at start of label / hint

This will allow us to create labels that start with a list item, such as this:

 <value>
* We would like to know how good or bad your health is TODAY.
* This scale is numbered from 0 to 100.
* 100 means the **best** health you can imagine
* 0 means the **worst** health you can imagine
* Please tap on the scale to indicate how your health is TODAY.
</value>

I agree such a label is weird though, so no problem if the decision is to not cater to this usage.

Avoid relying on a compiled JAR file from external PRs

So far, external contributions of changes to the ODK validate Jar files have been helpful, but it would be much healthier for the repository to have a different way for the JAR to be compiled (perhaps locally after installation)

Add external intent values populating

http://opendatakit.org/help/form-design/external-apps/

External intent launching is supported through appearance, but the group-of-values intent launching not. These need to be added to XLSForm.

Migration to Python 3?

Esteemed maintainers:

Guido, benevolent dictator for life, has renewed the call to transition projects from Python 2 to Python 3 at PyCon2015 (see https://youtu.be/G-uKNd5TSBw?t=1m21s). What are the thoughts of the pyxform team for doing this? Is it completely out of the question?

getting error while using pull data

Dear Team,

I have tried pull data from sample-preloading form provided by ODK website, getting below error.

Error occured during the loading of this form. It is recommended not to use this form for data entry until this is resolved.

Please contact [email protected] with the link to this page and the error message below:

FormLogicError: instance "hhplotdetails" does not exist in model

Add support for ODK Length

In ODK Aggregate 1.x, you can supply an attribute to the to specify the maximum number of characters used to store a string field; by default, the datastore layer limits strings to 255 characters or less. If a submission has data beyond this length, it is silently truncated upon submission.

Specifying a value less than 255 can significantly improve the storage efficiency of the database layer. You can also specify a value greater than 255, up to a maximum of about 16000 UTF-8 characters. The higher you set this value, the worse the storage efficiency within the datastore.

To set the string length:

In the header of the form definition, add a xmlns alias for odk (shown added as the last xmlns alias at the bottom of the tag):

xmlns:h="http://www.w3.org/1999/xhtml" xmlns:ev="http://www.w3.org/2001/xml-events" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:jr="http://openrosa.org/javarosa" xmlns:odk="http://www.opendatakit.org/xforms" ><h:head>

Then, in the bind, you can define an odk:length attribute to specify the length of a string field. Length can be specified for barcode, select1 and string fields. Multi-selects (select) fields have a 255 character maximum length for each selection value (and selection values must not contain whitespace); there are no limits to the number of choices that may be selected within a multi-select.

Do not url encode HTML namespaced tags

When I put the following in an XLSForm's label or hint

<h:font color="red">red </h:font>
<h:font color="#f58a1f">#f58a1f </h:font>
<h:p>
    <h:font face="monospace">monospace </h:font>
</h:p>

I get this in the XML:

<h:font color="red">red </h:font>
<h:font color="#f58a1f">#f58a1f </h:font>
<h:p>
    <h:font face="monospace">monospace </h:font>
</h:p>

Tags with the h namespace (<h:) should be passed through.

Bad XLSForm calculation breaks code, and why are errors not dealt with when using xls2xform.py as main?

Wonderful work so far on pyxform! I use pyxform to automate xform creation for my project. I have noticed that a bad calculate in XLSForm (a survey with a single line, e.g. {type: 'calculate', name: 'badcalc', label: 'This calculation fails', calculation:'because it doesn't know what to do'}) produces an error such as

Traceback (most recent call last):
  File "/path/to/package/pyxform/pyxform/xls2xform.py", line 65, in <module>
    warnings = xls2xform_convert(args.path_to_XLSForm, args.output_path)
  File "/path/to/package/pyxform/pyxform/xls2xform.py", line 22, in xls2xform_convert
     survey.print_xform_to_file(xform_path, validate=True, warnings=warnings)
  File "/path/to/package/pyxform/pyxform/survey.py", line 446, in print_xform_to_file
    warnings.extend(check_xform(path))
  File "/path/to/package/pyxform/pyxform/odk_validate/__init__.py", line 112, in check_xform
     m = 'ODK Validate Errors:\n' + _cleanup_errors(stderr)
  File "/path/to/package/pyxform/pyxform/odk_validate/__init__.py", line 86, in _cleanup_errors
return u'\n'.join(k)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcd in position 170: ordinal not in range(128)

This is not very informative. The problem is that the current code keeps python's place indicator of where the error is. This cannot be converted to unicode implicitly, so an error is thrown. I have made a fix to the code (locally), and now the error message says

ODK Validate Errors:
org.javarosa.xform.parse.XFormParseException: Invalid calculate for the bind attached to "${badcalc}" : Couldn't understand the expression starting at this point: because͎it doesn't know... in expression because it doesn't know what to do

>> XForm is invalid. See above for the errors.

Result: Invalid

But now the problem is that an ODK Validate error raises an unhandled Exception. A change to the main method (read: inside the if __name__ == '__main__': suite) in xls2xform.py to handle these exceptions would be pretty easy.

Nafundi's XLSForm Offline handles conversion errors, but does not handle the bad calculate. I would have thought that Nafundi's changes would make it back into the code, but I suppose not.

I would be happy to contribute what I have done to the project. What is the process to contribute/submit pull requests? What about creating test cases or checking that my new code does not break the old code?

Itemset from previous answers - select one from ${node}

Triggered by this question

It exposes the so-far-untapped XForm capability to create a choice list from previous answers (in a repeat).

It could have a wider implication by avoiding the more difficult indexed-repeat() or a combination of concat() and selected-at().

After discussion with @dorey we came up with the following initial syntax suggestion:

type	name	label	choice_filter
begin repeat	rep
text	name	Enter name
end repeat
select one from ${name}	choice	Choose	string-length(./name) > 0

<select1 ref="/data/choice">
   <label>Choose name</label>
   <itemset nodeset="/data/rep[string-length(./name) > 0]">
       <label ref="name"/>
       <value ref="name"/>
   </itemset>
</select1>

The ref attribute is the nodename of ${name} and the nodeset attribute is the parent of ${name} with an optional choice filter.

Add support for external instances

earlier discussion on ODK dev forum

• Support for both select_one_from_file [FILENAME] and select_multiple_from_file [FILENAME]
• The above types always create regular <select> or <select1> with a regular <itemset> and always add a single <instance id="" src="" /> to the model (unless that instance with that id already exists).
• Filenames can have either .csv or .xml extensions.
• instance id attribute = lowercased filename minus the extension: countries.csv => <instance id="countries">
• instance src attribute for XML sources gets jr://file/ prefix and for CSV sources gets jr://file-csv/ prefix (similar to jr://image, jr://video and jr://audio).
• the label ref of an itemset is always ref="label" because itext references couldn't work.

Examples

1a. XLSForm input (CSV file, no choice filter):

type	name	label	choice_filter
select_one_from_file countries.csv	country	Pick one country

1b. XForm output:

Very similar to a regular select_one with choice filter (but in this case the choice filter is empty)

<model>
    <instance>
        ....
    </instance>
    <instance id="cities" src="jr://file-csv/cities.csv" />
</model>
<h:body>
    <select1 ref="/myform/country">
        <label>Country</label>
        <itemset nodeset="instance('countries')/root/item">
            <value ref="name"/>
            <label ref="label"/>
        </itemset>
    </select1>
</h:body>

2a. XLSForm input (XML file, no choice filter):

type	name	label	choice_filter
select_one_from_file countries.xml	country	Pick one country

2b. XForm output:

Exactly the same syntax as used by CommCare for their many XML external instances

<model>
    <instance>
        ....
    </instance>
    <instance id="cities" src="jr://file/cities.xml" />
</model>
<h:body>
    <select1 ref="/myform/country">
        <label>Country</label>
        <itemset nodeset="instance('countries')/root/item">
            <value ref="name"/>
            <label ref="label"/>
        </itemset>
    </select1>
</h:body>

3a. XLSForm input (CSV file, with choice filter):

type	name	label	choice_filter
select_multiple_from_file cities.csv	city	Pick some cities	country = ${country}

3b. XForm output:

<model>
    <instance>
        ....
    </instance>
    <instance id="cities" src="jr://file-csv/cities.csv" />
</model>
<h:body>
    <select ref="/myform/country">
        <label>Country</label>
        <itemset nodeset="instance('cities')/root/item[country= /myform/country ]">
            <value ref="name"/>
            <label ref="label"/>
        </itemset>
    </select>
</h:body>

Notes:

• this feature is not compatible with ODK Collect (and vice versa select_external and search are not compatible with Enketo)
• the external data referred to in an XLSForm needs to always have a label column (that's an XLSForm restriction)
• csv column headers may not contain a colon (this not relevant to Pyxform but to the xlsform.org documentation)
• these XForms do not pass ODK Validate validation (I suspect it checks whether itemset nodeset paths exist), so to use this feature ODK Validate would have to be switched off or tweaked so it bypasses checks for paths that refer to external instances (i.e. that refer to instance(ID)/path/to/node when the targeted instance has a src attribute)

pyxform 0.9.22 validation not completing

What steps will reproduce the problem?
1.Install pyxform 0.9.22
2.Run converter
3.Conversion completes but validation does not

What is the expected output? What do you see instead?
Successful conversion and validation. Conversion to XML completes but validation does not.

Traceback (most recent call last):
File "C:\Python27\Lib\site-packages\pyxform\xls2xform.py", line 65, in <module

warnings = xls2xform_convert(args.path_to_XLSForm, args.output_path)

File "C:\Python27\Lib\site-packages\pyxform\xls2xform.py", line 22, in xls2xfo
rm_convert
survey.print_xform_to_file(xform_path, validate=True, warnings=warnings)
File "C:\Python27\Lib\site-packages\pyxform\survey.py", line 394, in print_xfo
rm_to_file
warnings.extend(check_xform(path))
File "C:\Python27\Lib\site-packages\pyxform\odk_validate__init__.py", line 96
, in check_xform
if not java_installed():
File "C:\Python27\Lib\site-packages\pyxform\odk_validate__init_.py", line 43
, in _java_installed
p = Popen(["which","java"], stdout=PIPE)
File "C:\Python27\lib\subprocess.py", line 711, in init
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 948, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified

What version of the product are you using? On what operating system?
pyxform 0.9.22 Windows 7 OS

Please provide any additional information below.
I could be wrong, I'm not a Unix expert; but it looks to me like this is an issue with a Unix specific command being included in pyxform which doesn't work on Windows. It is in pyxform/odk_validate/init.py, the _java_installed function, the "which" command submitted through Popen.

def _java_installed():
p = Popen(["which","java"], stdout=PIPE)
return len(p.stdout.readlines()) != 0

When I comment out the call for this function, converter and validation both work fine.

Mixed variable and Urdu label result in misordered layout

I want to have the following
Q6. صوبہ ؟ ${province} کیا

I put this in the XLSForm:

It shows up in the XML like this:

It shows up this in ODK Collect like this:

Note Accra (the province) is moved to the end...

Add relative referencing

The latest Javarosa version for ODK Collect now has

current()/../../ref syntax. To do relative referencing of fields in an XLSForm. One could theoretically replace all the ${fieldname} expansions with this syntax and then we would have reliable referencing of fields within the current repeat group. This is not documented anywhere (yet).

Add ability to add additional namespaces

Referring to Esri's additions and to custom features I am working on, it would be great if we could add additional space-separated prefix="namespace" e.g. in the settings sheet under a namespaces column that result in the corresponding xmlns attributes on the html element.

namespaces
esri="http://esri.com/xforms" enk="http://enketo.org/xforms" naf="http://nafundi.com/xforms"

This would allow us to use survey columns such as: bind::esri:fieldType and bind::nk2:myattribute and not be tempted to either use no namespace or an incorrect (but already existing) one.

So the above would ensure the following namespace declarations are in the XForm output:

<h:html .... 
    xmlns:esri="http://esri.com/xforms" 
    xmlns:enk="http://enketo.org/xforms" 
    xmlns:naf="http://nafundi.com/xforms"
>

Forms with choice filters crash at has_external_choices

If a form has a choice_filter, the check for has_external_choices fails with

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pyxform/xls2xform.py", line 65, in <module>
    warnings = xls2xform_convert(args.path_to_XLSForm, args.output_path)
  File "/usr/local/lib/python2.7/site-packages/pyxform/xls2xform.py", line 24, in xls2xform_convert
    if has_external_choices(json_survey):
  File "/usr/local/lib/python2.7/site-packages/pyxform/utils.py", line 115, in has_external_choices
    elif has_external_choices(v):
  File "/usr/local/lib/python2.7/site-packages/pyxform/utils.py", line 113, in has_external_choices
    if k == u"type" and v.startswith(u"select one external"):
AttributeError: 'list' object has no attribute 'startswith'

The problem is that choices are also represented as k=type. Easiest fix is to make sure v is a list.

if k == u"type" and isinstance(v,basestring) and v.startswith(u"select one external"):

Prevent choice filter from creating additional languages in form

See sample xlsform here. This is modified xlsform that is able to send data into OpenMRS via the openmrs_code column in choices sheet. The choice filter causes openmrs_code to show up as an extra language in webform view.

@ukanga

Add non-string calculation types

EDIT: see #70 (comment) for the latest proposed approach.

Initial bug reported at https://groups.google.com/d/msg/opendatakit/eYFsVCLet1U/Wb4wZumGBAAJ

"The problem seems to be that the total_feld_calc field is stored as a "string" field, so the subsequent sum(${total_field_calc}) will fail with a type conversion failure."

Invalid expression causes an exception

An invalid expression, such as $(new_post_installNo)='2' (incorrect parentheses) causes an exception such as -

Traceback (most recent call last):
  File "/Users/elvin/py/pyxform/xls2xform.py", line 94, in <module>
    main_cli()
  File "/Users/elvin/py/pyxform/xls2xform.py", line 81, in main_cli
    response['message'] = str(e)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u034e' in position 209: ordinal not in range(128)

Remove trailing slash from the uri of OpenRosa features namespace

Change http://openrosa.org/xforms/ to http://openrosa.org/xforms

This was raised by @mitchellsundt

Better error message when list name = type

When there is a list name of 'type' in the choices sheet, the error reads something like: 'list' object has no attribute 'startswith' (ref onaio/onadata#762)

remove type="cascading_select"?

I just got a user report with a form that used type="cascading_select".

I was not familiar with it. Is this old code? Maybe it should be removed?

Hint text for groups is ignored

Hint text specified for groups is ignore instead of creating a <hint> element.

This appears to be the same issue as #73, so resolving this issue may also resolve #73

Strip spaces from headers

When headers name or type are given with preceding or following spaces (ie, as name, type [1]), pyxform is not striping them. This causes a lot of confusing among newbies, especially, and in general. In Excel, it is incredibly hard to tell if there is a trailing space or not.
The same may also be true of the different sheets (survey and choices). While we are at it, the headers + sheet names should also be auto-lowercased.

Note, this may be a relatively easy fix for someone new to pyxform. As such, I added a "get-started-issues" label to this issue. We should add language to a "How to Contribute" section on pyxform somewhere to a list of issues tagged with this label.

[1] - Oh, you can't see the difference? That is because its pretty impossible to, by just looking at things. Or even github markdown strips spaces. One of the two :-P

Add undocumented functionality in the XML

• If you specify rows="5" for a text input field, it is equivalent to a multiline="5" attribute. http://opendatakit.org/help/form-design/examples/#string_prompts
• You can customize button text in 3rd party app launching prompts via itext alternate names http://opendatakit.org/help/form-design/examples/#string_3rd_party
• You can specify an accuracyThreshold attribute on a geopoint prompt. This sets the auto-accept threshold for the geopoint capture.
• You can also specify autoplay="audio" or autoplay="video" to automatically play audio or video prompts when the question is displayed http://opendatakit.org/help/form-design/examples/#presentation

Handle unicode characters in external_sheet

'ascii' codec can't encode character u'\u2019' in position 12: ordinal not in range(128)

Version 0.9.23

Includes:

• select_one_external support
• configurable namespaces (in settings sheet- namespaces column)

Duplicate names for questions across levels

Hi,

What is the rationale to allow duplicate names for questions across levels?
e.g a date question at the top level can have the same name as a date question within a group.

Is this something that will be restricted sometime in the future?