truelite / python-a38 Goto Github PK

Ciao, il link nel readme non esiste più

Hello, thank you for this very useful project!
Would you consider Pull Requests adding Github Actions to automatically run tests with a badge to monitor the outcome?

Under Windows 10 I detect an issue in the way you generate the temporary html file to transformed. In particular, the temporary file is not closed and the wkhtmltopdf process cannot access to it for reading.

Exception

RuntimeError: ('%s exited with error %d: stderr: %s', 'C:\\Program Files\\wkhtmltopdf\\bin\\wkhtmltopdf.EXE', 1, b'Loading pages (1/6)\r\n[> ] 0%\r[======> ] 10%\rError: Failed to load file:///C:/Users/xxxxxxx/AppData/Local/Temp/tmpx8xa1dir.html, with network status code 201 and http status code 0 - Error opening C:/Users/xxxxxxx/AppData/Local/Temp/tmpx8xa1dir.html: Impossibile accedere al file. Il file \xe8 utilizzato da un altro processo.\r\nError: Failed loading page file:///C:.....

My personal quick fix, but it's needed a better error handling:

fd = tempfile.NamedTemporaryFile("wb", suffix=".html", delete = False)
html.write(fd)
fd.flush()
name = fd.name
fd.close()
            
res = subprocess.run([wkhtmltopdf, name, output_file], stdin=subprocess.DEVNULL, capture_output=True)
os.remove(name)

Currently the tests in tests/test_p7m.py stop working after the test signature expires.

I'm not aware of a way to generate a CAdES signature using a custom CA and certificates that do not expire, so for now we'll have to regenerate that test fixture every few years. I've documented the step-by-step instructions in the file.

However, to prevent sudden test failures, we can also change the test to check if the signature is expired, and skip instead of failing.

There are still two biggish open MR/feature branches, but there are a few fixed that landed in main that deserve a release, expecially the updated test certificate since the last one expired, causing a test to fail

Given the long names of a38 elements, it would be quite easy to make a mistake like dati_generali_documento.date = mydate and have it silently do nothing (the correct attribute would be data).

It might be that with a clever use of __slots__, possibly in the metaclass, one can disable creation of new members in a38 models, meaning that something such as this could be caught

According to AgenziaEntrate.gov.it / Specifiche Tecniche multiple DatiRitenuta tags are possible in DatiGeneraliDocumento since version 1.6 (4.5.2020)

bandit suggests to use defusedxml and it has a point. Python's documentation suggests that, too.

Let's do that, possibly falling back to the standard library with a warning if defusedxml is not installed

The two README.md links to the xsl files from fatturapa.gov.it:

• https://www.fatturapa.gov.it/export/fatturazione/sdi/fatturapa/v1.2/fatturaordinaria_v1.2.xsl
• https://www.fatturapa.gov.it/export/fatturazione/sdi/fatturapa/v1.2/fatturapa_v1.2.xsl

are currently broken (404).

Now that a38tool can save and load YAML, it's easy to make an a38tool edit command that converts a fattura to YAML or Python, opens it in an editor, and then writes it back as XML.

It could be a very convenient way to manually fixup an XML.

A package build with

python setup.py build sdist

produces the following error when installed using pip install dist/a38-0.1.5.tar.gz

Processing ./dist/a38-0.1.5.tar.gz
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [8 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-req-build-krekzuva/setup.py", line 25, in <module>
          install_requires=parse_requirements("requirements-lib.txt"),
        File "/tmp/pip-req-build-krekzuva/setup.py", line 9, in parse_requirements
          line_iter = (line.strip() for line in open(filename))
      FileNotFoundError: [Errno 2] No such file or directory: 'requirements-lib.txt'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

It is possible to manually perform extra validation of a fattura using the official XML Schema Definition with xmlstarlet:

xmlstarlet val  -e -s  doc/Schema_del_file_xml_FatturaPA_versione_1.2.1.xsd  filename.xml

This could possibly be integrated into a38/

The xsd used is downloaded with the download-docs script in the a38 git repository

If you define a document with dati_bollo, adding a document's linea for it, aliquota_iva must be 0 (art. 15 del D.P.R. n. 633/1972), as well as natura need to be set with a code (here N1¹ code) .
We get an error, though:

$ python main.py 
fattura_elettronica_body.0.dati_beni_servizi.natura: [00429] field is empty while aliquota_iva is zero

To reproduce, update your example in README like that:

1. build a document with dati_bollo in DatiGeneraliDocumento

body.dati_generali.dati_generali_documento = DatiGeneraliDocumento(
    tipo_documento="TD01",
    divisa="EUR",
    data=datetime.date.today(),
    numero=bill_number,
    causale=["Test billing"],
    dati_bollo=DatiBollo(
        bollo_virtuale="SI",
        importo_bollo="2",
    )
)

2. Add a line in invoice, with a "Bollo" 2€ record, with 0 aliquota_iva and a natura code

...
body.dati_beni_servizi.add_dettaglio_linee(
    descrizione="Bollo", quantita=1, unita_misura="EUR",
    prezzo_unitario="2", aliquota_iva="0.00", natura="N1",
)

3. Check out the resulting xml. This is supposed to build nodes like that:

<DatiBollo>
   <BolloVirtuale>SI</BolloVirtuale> 
   <ImportoBollo>2.00</ImportoBollo>
</DatiBollo>
...
<NumeroLinea>3</NumeroLinea>
    <Descrizione>Bollo</Descrizione>
    <Quantita>1.00</Quantita>
    <UnitaMisura>EUR</UnitaMisura>
    <PrezzoUnitario>2.00</PrezzoUnitario>
    <PrezzoTotale>2.00</PrezzoTotale>
    <AliquotaIVA>0.00</AliquotaIVA>
    <Natura>N1</Natura>
</DettaglioLinee>
...
<DatiRiepilogo>
    <AliquotaIVA>0.00</AliquotaIVA>
    <Natura>N1</Natura>
   <ImponibileImporto>2.00</ImponibileImporto>
   <Imposta>0.00</Imposta>
   <EsigibilitaIVA>I</EsigibilitaIVA>
</DatiRiepilogo>

And it works, it's built correctly. But we get an apparently false-negative:

$ python main.py 
fattura_elettronica_body.0.dati_beni_servizi.natura: [00429] field is empty while aliquota_iva is zero

This happens on validation stage, on your example, when you print out if errors occour:

# file: main.py
if res.errors:
    for e in res.errors:
        print(str(e), file=sys.stderr)

The reason is because it expects DatiRiepilogo.natura to be instantiated, while it's not:

# file: fattura.py:447
    def validate_model(self, validation):
        super().validate_model(validation)
        if self.aliquota_iva == 0 and self.natura is None: # <--- this check fails
            validation.add_error(
                self._meta["natura"], "field is empty while aliquota_iva is zero", code="00429")

To fix it, in fattura.py DatiBeniServizi().build_dati_riepilogo should be defined natura var and it has to be passed to the model, like that:

# file: fattura.py:496
        self.dati_riepilogo = []
        for aliquota, linee in sorted(by_aliquota.items()):
            natura = linee[0].natura # <--- setting new var to pass on DatiRiepilogo() model
            imponibile = sum(linea.prezzo_totale for linea in linee)
            imposta = imponibile * aliquota / 100
            self.dati_riepilogo.append(
                    DatiRiepilogo(
                        aliquota_iva=aliquota, imponibile_importo=imponibile,
                        imposta=imposta, esigibilita_iva="I", natura=natura)) # <-- here we go :)

Now it works, keeping to build a valid xml, without any failed validation:

$ python main.py 
$ ls
bin  include  IT01234567890_00001.xml  lib  lib64  main.py  pyvenv.cfg  README.md  requirements.txt  stubs  tests 

¹: see https://www.agendadigitale.eu/documenti/esperto-risponde/fattura-elettronica-ecco-come-indicare-il-bollo/

Buongiorno , premetto che sto usando la vostra procedura per generare la fattura elettronica da un database sql e la trovo fantastica, ho un solo problema, se genero la fattura con i codici iva non ci sono problemi e questa è la riga che uso per aggiungere i dettagli:

body.dati_beni_servizi.add_dettaglio_linee(
descrizione="Test item", quantita=2, unita_misura="kg",
prezzo_unitario="25.50", aliquota_iva="22.00")

anche controllando l'xml me lo da corretto

mentre se metto l'aliquota iva a "0" indicando la natura:

body.dati_beni_servizi.add_dettaglio_linee(
descrizione="Other item", quantita=1, unita_misura="kg",
prezzo_unitario="15.50", aliquota_iva="0.00", natura="N3.1")

mi da questo errore :
fattura_elettronica_body.0.dati_beni_servizi.natura: [00429] field is empty while aliquota_iva is zero

e anche da controllo formale me la scarta

potete gentilmente dirmi dove sto sbagliando?

a38tool fails with an uncaught exception:

$ a38tool --help
uncaught exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/a38-0.1.3-py3.6.egg/EGG-INFO/scripts/a38tool", line 360, in
File "/usr/local/lib/python3.6/dist-packages/a38-0.1.3-py3.6.egg/EGG-INFO/scripts/a38tool", line 330, in main
File "/usr/lib/python3.6/argparse.py", line 1716, in add_subparsers
action = parsers_class(option_strings=[], **kwargs)
TypeError: init() got an unexpected keyword argument 'required'

Am I missing something?

The download-docs script is currently broken:

$ ./download-docs 
2021-10-03 21:36:26,305 ERROR uncaught exception
Traceback (most recent call last):
  File "/tmp/python-a38/./download-docs", line 100, in <module>
    main()
  File "/tmp/python-a38/./download-docs", line 94, in main
    download("https://www.fatturapa.gov.it/export/fatturazione/it/normativa/f-2.htm")
  File "/tmp/python-a38/./download-docs", line 57, in download
    for el in get_urls(index_url):
  File "/tmp/python-a38/./download-docs", line 38, in get_urls
    for a in root.iter("a"):
AttributeError: 'NoneType' object has no attribute 'iter'

This may be related to #10, as the script may be failing to retrieve online resources.

The rounding of to_xml does not follow the official rules

– per difetto se la terza cifra dopo la virgola è compresa tra 0 e 4;
– per eccesso se la terza cifra dopo la virgola è compresa tra 5 e 9.

For example, the value 0.385 should be rounded to 0.39 instead of 0.38.

Example

import a38.fattura as a38

fattura = a38.FatturaPrivati12()
body = fattura.fattura_elettronica_body[0]
body.dati_beni_servizi.add_dettaglio_linee(
    descrizione="Item 1", prezzo_unitario="0.35", aliquota_iva=10, unita_misura='pz', quantita=1
)
body.dati_beni_servizi.build_dati_riepilogo()
body.build_importo_totale_documento()

fattura.fattura_elettronica_body[0].dati_generali.dati_generali_documento.importo_totale_documento  # is Decimal('0.385')
fattura.build_etree().getroot().find('.//ImportoTotaleDocumento').text                              # should be 0.39

wkhtmltopdf version 0.12.6 blocks by default the local access to file.
It's needed to to add the option --enable-local-file-access to have a successfull transformation of html file to pdf

can be changed in

res = subprocess.run([wkhtmltopdf, "--enable-local-file-access", fd.name, output_file], stdin=subprocess.DEVNULL, capture_output=True)

Affecting a38 version 0.1.2: when a fattura contains an attachment in base64 form under tag

<ns3:FatturaElettronica xmlns:ns3="http://ivaservizi.agenziaentrate.gov.it/docs/xsd/fatture/v1.2" xmlns:ns2="http://www.w3.org/2000/09/xmldsig#" versione="FPR12">
<FatturaElettronicaBody><Allegati><**Attachment**>

and we try to convert into PDF with wkhtmltopdf the following error is raised:

`ERROR uncaught exception
Traceback (most recent call last):
  File "/usr/local/bin/a38tool", line 350, in <module>
    main()
  File "/usr/local/bin/a38tool", line 343, in main
    res = app.run()
  File "/usr/local/bin/a38tool", line 247, in run
    self.render(f, output)
  File "/usr/local/bin/a38tool", line 288, in render
    self.transform.to_pdf(self.wkhtmltopdf, f, output)
  File "/usr/local/lib/python3.8/dist-packages/a38/render.py", line 32, in to_pdf
    html = self(f)
  File "/usr/local/lib/python3.8/dist-packages/a38/render.py", line 21, in __call__
    tree = f.build_etree(lxml=True)
  File "/usr/local/lib/python3.8/dist-packages/a38/fattura.py", line 685, in build_etree
    self.to_xml(builder)
  File "/usr/local/lib/python3.8/dist-packages/a38/fattura.py", line 671, in to_xml
    field.to_xml(b1, getattr(self, name))
  File "/usr/local/lib/python3.8/dist-packages/a38/fields.py", line 669, in to_xml
    val.to_xml(builder)
  File "/usr/local/lib/python3.8/dist-packages/a38/models.py", line 154, in to_xml
    field.to_xml(b, getattr(self, name))
  File "/usr/local/lib/python3.8/dist-packages/a38/fields.py", line 669, in to_xml
    val.to_xml(builder)
  File "/usr/local/lib/python3.8/dist-packages/a38/models.py", line 154, in to_xml
    field.to_xml(b, getattr(self, name))
  File "/usr/local/lib/python3.8/dist-packages/a38/fields.py", line 115, in to_xml
    builder.add(self.get_xmltag(), self.to_str(value))
  File "/usr/local/lib/python3.8/dist-packages/a38/builder.py", line 27, in add
    self.etreebuilder.end(tag)
  File "src/lxml/saxparser.pxi", line 836, in lxml.etree.TreeBuilder.end
  File "src/lxml/saxparser.pxi", line 771, in lxml.etree.TreeBuilder._handleSaxEnd
  File "src/lxml/saxparser.pxi", line 740, in lxml.etree.TreeBuilder._flush
TypeError: sequence item 0: expected str instance, bytes found`

I need to write some fields with UTF-8 characters e.g. <Denominazione>Güügle</Denominazione>.
I've taken a valid XML file with some values like the above after having retrieved it from the AdE website (Agenzia Delle Entrate). This XML file is SDI validated.

1. I would like to store the above encoding in the invoice XML file with some code using the a38 library.
2. I would like to generate a PDF from the XML file using the command a38tool pdf and the usual FoglioStileAssoSoftware.xsl from AssoSoftware.

However these two things don't seem to work well together.

Take this code snippet where it seems I could use the a38 XML builder instead of LXML:

import os

import a38.fattura

f = a38.fattura.FatturaPrivati12(
    fattura_elettronica_header=a38.fattura.FatturaElettronicaHeader(
        cessionario_committente=a38.fattura.CessionarioCommittente(
            dati_anagrafici=a38.fattura.DatiAnagraficiCessionarioCommittente(
                anagrafica=a38.fattura.Anagrafica(denominazione="Güügle"),
            )
        )
    )
)

real_path = os.path.realpath(__file__)
dir_path = os.path.dirname(real_path)

# this renders as `<Denominazione>G&#xFC;&#xFC;gle</Denominazione>`
filename_lxml = f"{dir_path}/foo-invoice-lxml.xml"
tree = f.build_etree(lxml=True)
with open(filename_lxml, "wb") as out:
    tree.write(out, encoding="utf8")
    # tree.write(out)

# THIS WORKS!!!
# this renders as `<Denominazione>Güügle</Denominazione>`
filename_other = f"{dir_path}/foo-invoice-other.xml"
tree = f.build_etree()
with open(filename_other, "wb") as out:
    tree.write(out, encoding="utf8")
    # tree.write(out)

This way I am able to store a XML file to the file system containing <Denominazione>Güügle</Denominazione>. All good.

When I invoke a command like:

 a38tool pdf \
         -f FoglioStileAssoSoftware.xsl \
         -o foo-invoice-other.pdf \
         foo-invoice-other.xml # <--- this is the XML file I generated with the Python code snippet above

Then I get this error:

ERROR uncaught exception
Traceback (most recent call last):
  File "<<...>>/bin/a38tool", line 360, in <module>
    main()
  File "<<...>>/bin/a38tool", line 353, in main
    res = app.run()
  File "<<...>>/bin/a38tool", line 256, in run
    f = self.load_fattura(pathname)
  File "<<...>>/bin/a38tool", line 31, in load_fattura
    tree = ET.parse(pathname)
  File "/usr/lib/python3.8/xml/etree/ElementTree.py", line 1202, in parse
    tree.parse(source, parser)
  File "/usr/lib/python3.8/xml/etree/ElementTree.py", line 595, in parse
    self._root = parser._parse_whole(source)
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 42, column 36

That line 42, column 36 in the error is the umlaut in Güügle.

How can I both:

• generate an XML file with UTF8 characters using the a38 library?
• and generate a PDF file with the a38tool command?

Some context:

$ pip show lxml | grep Version
Version: 4.7.1
$ pip show a38 | grep Version
Version: 0.1.3
$ python --version
Python 3.8.10

One final thing: I did not get a valid invoice from the SDI (AdE) with the first row as <?xml version="1.0" encoding="utf-8"?> although it was indeed containing some UTF-8 encoded characters like above.

I found out from the AssoSoftware FAQ http://www.assosoftware.it/faq?catid=0&limit=10&start=50 that they recommend this encoding information to be the first line of the XML file. I am not sure if this is an issue with AdE/SDI or a too flexible interpetation, anyway here's the snippet:

FORMATO DI CODIFICA
22/02/2019

Le specifiche tecniche SDI della fattura XML non indicano obbligatoriamente l'utilizzo della codifica UTF-8 tuttavia è consigliata per una corretta interpretazione dei dati inseriti. Sono da evitare altre codifiche (es. ISO-8859, CP1252, ecc..) che obbligano a conversioni dei dati in fase di lettura del file.
In apertura il file xml dovrebbe riportare come prima riga la versione del file xml e la codifica utf-8:

<?xml version="1.0" encoding="utf-8"?>

Hello, another useful thing would be to add a variety of Pip packages that help in making the source code looking the same regardless the project and check things like PEP8, static code analysis, type checkers and vulnerability checkers.

There's a variety of Pip projects that cover the topics above, would you be interested in a Pull Request to add all these tools and checks? It would come handy in the CI pipeline as well.

Sometimes fatture elettroniche have attachments, e.g.

[...]
    <Allegati>
      <NomeAttachment>someattachment.pdf</NomeAttachment>
      <DescrizioneAttachment>attachment description</DescrizioneAttachment>
      <Attachment>[encoded data]</Attachment>
    </Allegati>
  </FatturaElettronicaBody>
</ns2:FatturaElettronica>

It would be useful to list and extract them with a38.

As per here (or similar docs from AdE): https://www.agenziaentrate.gov.it/portale/documents/20143/451259/Guida_compilazione-FE_25+11+20.pdf/e811fe3d-332d-5b51-5990-10d7e4641164

There's this new value for the "natura" field: N2.1 (and N2.2 as well actually).

N2.1 – Operazioni non soggette ad IVA ai sensi degli artt. da 7 a 7-septies del d.P.R. n.
633/72

If I validate a XML fattura file containing N2.1 (that passed the AdE SDI validation), then I see this:

fattura_elettronica_body.0.dati_beni_servizi.dettaglio_linee.0.natura: 'N2.1' is not a valid choice for this field
fattura_elettronica_body.0.dati_beni_servizi.dettaglio_linee.0.natura: 'N2.1' should be no more than 2 characters long
...
fattura_elettronica_body.0.dati_beni_servizi.dati_riepilogo.0.natura: 'N2.1' is not a valid choice for this field
fattura_elettronica_body.0.dati_beni_servizi.dati_riepilogo.0.natura: 'N2.1' should be no more than 2 characters long

I checked the code base and found a few places where these new strings could be added:

$ grep -nrIi '"N2"' <<...>>/site-packages/a38/
<<...>>/site-packages/a38/fattura_semplificata.py:95:    natura = fields.StringField(length=2, null=True, choices=("N1", "N2", "N3", "N4", "N5", "N6", "N7"))
<<...>>/site-packages/a38/fattura.py:280:    natura = fields.StringField(length=2, choices=("N1", "N2", "N3", "N4", "N5", "N6", "N7"), null=True)
<<...>>/site-packages/a38/fattura.py:357:    natura = fields.StringField(length=2, null=True, choices=("N1", "N2", "N3", "N4", "N5", "N6", "N7"))
<<...>>/site-packages/a38/fattura.py:377:    natura = fields.StringField(length=2, null=True, choices=("N1", "N2", "N3", "N4", "N5", "N6", "N7"))

Is there anything else that should be amended? I could provide a PR if that's fine. Thanks 🙏