arelle / arelle

Arelle open source XBRL platform

Arelle

Table of Contents

• Arelle
  • Table of Contents
  • Description
  • Documentation
  • Features
  • Need Support?
    • Arelle Within Other Products
    • EdgarRenderer or EFM Validations
    • Installing or Running Arelle
    • Bug Report or Feature Request
    • Security Vulnerabilities
    • How-To and General XBRL Questions
    • Email
  • How To Contribute
  • License

Description

Arelle is an end-to-end open source XBRL platform, which provides the XBRL community with an easy to use set of tools. It supports XBRL and its extension features in an extensible manner. It does this in a compact yet robust framework that can be used as a desktop application and can be integrated with other applications and languages utilizing its web service, command line interface, and Python API.

Documentation

Need help with Arelle? Go check out our documentation.

Features

• Fully-featured XBRL processor with GUI, CLI, Python API and Web Service API.
• Support for the XBRL Standard, including:
  • XBRL v2.1 and XBRL Dimensions v1.0
  • XBRL Formula v1.0
  • Taxonomy Packages v1.0
  • xBRL-JSON v1.0 and xBRL-CSV v1.0
  • Inline XBRL v1.1
  • Units Registry v1.0
• Certified by XBRL International as a Validating Processor.
• Support for filing programme validation rules:
  • Edgar Filer Manual validation (US SEC)
  • ESEF Reporting Manual (EU)
  • HMRC (UK)
  • CIPC (South Africa)
  • FERC (US Federal Energy Regulatory Commission)
• Integrated support for Arelle Inline XBRL Viewer.
• Extensible plugin architecture.
• Support for XF text-based Formula and XULE validation rules.
• The Web Service API allows XBRL integration with applications, such as those in Excel, Java or Oracle.
• Instance creation is supported using forms defined by the table linkbase.
• Support for reading/monitoring US SEC XBRL RSS feeds (RSS Watch).

Need Support?

Whether you've found a bug, need help with installation, have a feature request, or want to know how to use Arelle, we can help! Here's a quick guide:

When reporting issues it's important to include as much information as possible:

• what version of Arelle are you using?
• how are you using Arelle (GUI, command line, web server, or the Python API?)
• what operating system (Windows, macOS, Ubuntu, etc.) are you using?
• what plugins if any do you have enabled?
• can you provide an XBRL report that recreates the issue?
• what's the diagnostics output (arelleCmdLine.exe --diagnostics) on your system?

Arelle Within Other Products

A number of service providers embed Arelle within their XBRL products and tools. If you're having an issue with Arelle within one of these offerings please contact the developer of that tool for support or first verify that you have the same issue when using Arelle directly. Most issues in these situations are caused by the tool using an old version of Arelle or not running a valid command.

EdgarRenderer or EFM Validations

The SEC develops and maintains the EdgarRenderer and EFM validation plugins. Please report issues with these plugins directly to the SEC ([email protected]).

Installing or Running Arelle

Most installation and startup issues can be resolved by downloading the latest version of Arelle and performing a clean install. If that doesn't resolve the problem for you, please report a bug.

Bug Report or Feature Request

Please use the GitHub issue tracker if you'd like to suggest a new feature or report a bug.

Before opening a new issue, please:

• Check that the issue has not already been reported.
• Check that the issue has not already been fixed in the latest release.
• Be clear and precise (do not prose, but name functions and commands exactly).
• For bug reports include the version of Arelle you're using.

Security Vulnerabilities

Identified a security concern? Email the Arelle team ([email protected]) so we can resolve the issue and make sure service providers and authorities who use Arelle in production are prepared to update and apply security patches before notifying the general public.

How-To and General XBRL Questions

Have a question that isn't covered by the documentation? Join our Arelle Google Group and start a conversation with the Arelle team and community of experts.

Email

The Arelle team can also be reached by email ([email protected]) for issues that aren't a good fit for the other support channels. However, please note that you will likely receive a faster response if you open a GitHub issue or start a new conversation in the Arelle Google Group where the Arelle team is active and other people within the community can also see and respond to your message.

How To Contribute

Interested in contributing to Arelle? Awesome! Make sure to review our contribution guidelines.

👥 Contributors

Join our community and become a contributor today! 🚀

License

Apache License 2.0

arelle's Issues

Mac GUI crashes upon load Table Linkbase

Mac OS App version: 2022-08-08

Tried to load the following table linkbase into the Mac App:
EIOPA_SolvencyII_XBRL_Taxonomy_2.8.0/eiopa.europa.eu/eu/xbrl/s2md/fws/solvency/solvency2/2023-07-15/tab/e.02.01.16.01/e.02.01.16.01.xsd.

After loading the GUI crashes (no error message).

I cloned the Arelle repository (master branch) and ran the runMacGUI.sh with environment variable: TKTABLE_LIBRARY=/Arelle/libs/macos/Tktable2.11.
Afterwards loaded the same table linkbase which causes no errors in the local Mac GUI.

Could you please look into the downloadable mac distribution?
In case of any questions please let me know.

Incorrect release number from pip build

What happened?

When I use pip install git+https://[email protected]/arelle/arelle.git@master#egg=Arelle-release I get the right commit, but the wrong build number

Please let me know if there's a different command I should be using to install Arelle.

FWIW, unsure on the OS question; this is a Jupyter notebook - platform.uname() is system='Linux' version='#202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022', machine='x86_64'

Documents

No response

If running from the command line, what command did you run?

pip install git+https://[email protected]/arelle/arelle.git@master#egg=Arelle-release

Interface

Python library (pip install)

Version

964f5b1 <> 2020.05.10 (does it?)

Download

pip install

Operating System

Linux (Ubuntu 20.04)

Move packaging config to pyproject.toml

Modern python packaging best practices are to use pyproject.toml (or setup.cfg) to configure builds and avoid arbitrary code execution in setup.py. cxFreeze can't be configured from pyproject.toml (or setup.cfg), but the rest of the config should be able to be moved into pyproject.toml.

Are type annotations allowed in pull requests?

And if so, which would be the lowest python version they should be compatible with?

(3.9 allows list[str] instead of typing.List[str]; 3.10 allows str | None instead of Optional[str], etc.)

Type annotations allow static type checking to reduce the probability of bugs but also provide helpful indications in text editors that exploit them:

XmlUtil.innerText() nested span element whitespace

Problem: the ixt-sec:durwords transform is not raising an error for:

<ix:nonNumeric contextRef="c" format="ixt-sec:durwordsen" name="foo:bar" id="f1">5<span style="letter-spacing:-0.7pt;"> </span>years</ix:nonNumeric>

The element's inner text should not retain whitespace in this case.

(It would require <span style="letter-spacing:-0.7pt;">&#xA0;</span> to work properly. Adding white-space: pre; to the span style with a space character doesn't seem to work in other products.)

Related issue: ESEF text block discussions on XBRL.EU's BPTG calls.

References: mozilla W3C

Update python copyright headers

Mark V Systems is no more. Update python file copyright notices to Workiva.

Question regarding the Taxonomy package is not a zip file-error

While parsing ESEF-files, I'm getting the Taxonomy package is not a zip file below. I'm not convinced this is a bug as I note the test suite is based on zip-files, leading me to believe the XBRL-packages are expected to be in that format.

I couldn't get zip-files to work (the catalog.xml file wasn't parsed correctly), so I've unzipped and instead load the compressed file's contents. It works as expected so the Taxonomy package is not a zip file error is a bit confusing. It's logged here and then the code proceeds to parse the files.

Wouldn't it make more sense to be silent until the IOError error is raised in the end?

I'm also getting the tpe:invalidMetaDataFile error in every file but that seems to be due to using an XSD-file that hasn't been updated at the source for a while.

2022-07-21 15:33:39,914 [tpe:invalidArchiveFormat] Taxonomy package is not a zip file. - catalog.xml 

2022-07-21 15:33:39,935 [tpe:invalidMetaDataFile] Taxonomy package file syntax error Element '{urn:oasis:names:tc:entity:xmlns:xml:catalog}catalog': No matching global declaration available for the validation root., line 1 - catalog.xml 

2022-07-21 15:33:39,939 [tpe:invalidArchiveFormat] Taxonomy package is not a zip file. - taxonomyPackage.xml 

Please rebase lxml into master

Many git or github operations are based on 'master' by default (github homepage, sending a github pull request, creating a branch, diffing code, etc.). It seems 'master' is not active any more, when it ought to be the active branch ('lxml' is the active branch).

Please rebase 'lxml' into 'master'.

Validation performance issues with duplicated facts

There are some performance issues when validating with duplicated facts.

Step to reproduce:

• Add the CBSO taxonomy package (in annex)
• Open the xbrl file (in annex)
• Validate

It can be tested with the rule oa_06.05.1_0009 for example.

From my analysis, it's linked to https://github.com/Arelle/Arelle/blob/master/arelle/FormulaEvaluator.py#L1300, which at the end will make a loop over 1 953 158 evaluationResults here: https://github.com/Arelle/Arelle/blob/master/arelle/FormulaEvaluator.py#L437
Removing duplicate will reduce this list to 29.
Only 31 duplicated facts are removed between the 2 instances.

Note that the problem of the duplication of fact is in an external tools, and not linked to arelle, only the validation is impacted,

Taxonomy: nbb-cbso-22.18.19.zip
Report with duplicate: report_duplicate.zip
Report without duplicate: report_without_duplicate.zip

It may be linked to #217 since it's performance related.

Tested on current master (55cb2ca)

ESEF validation changes according to the ESEF reporting manual 2022

What should we change and why?

Dear Arelle team,

ESMA released a new 2022 reporting manual at the end of August 2022 (https://www.esma.europa.eu/file/24176/download?token=1z_i7DCc). There are many changes, but at least some of them don't have any technical implications for the processor (or the ESEF validation plugin).

Please find an overview here, which I hope will help you to focus on the technical aspects of the validation. Of course feel free to let me know about any disagreement with my valuation. Also, unfortunately I was not able to verify all of the below rules and some may need some discussion.

• Guidance 1.0.1 [new] – no implications for technical valdation
• Guidance 1.0.2 [new] – no implications for technical valdation
• Guidance 1.1.2 [updated] – no implications for technical valdation
• Guidance 1.3.3 [updated] – no implications for technical valdation
• Guidance 1.6.1 [updated] – I assume the severty level should be decreased from error to warning and the message should be changed "[ESEF.RTS.Annex.IV.Par.4.2.monetaryConceptWithoutBalance] Extension monetary concepts SHOULD provide balance attribute:"
• Guidance 1.9.1 [new] – no implications for technical valdation
• Guidance 1.9.2 [new] – no implications for technical valdation
• Guidance 1.9.3 [new] – no implications for technical valdation
• Guidance 2.2.4 [updated] – I am unsure if this needs adaption of the technical validation.
• Guidance 2.2.5 [new] – no implications for technical valdation
• Guidance 2.2.6 [new] – Concepts of type nonnum:textBlockItemType MUST be escaped when its content contains a < (U+003C) and/or & (U+0026) character (I am unsure if there are no additional "invalid" characters in XML, but ESMA clearly only states these two)
• Guidance 2.5.1 [updated] – Check for [ESEF.2.5.1.executableCodePresent] must be extended to check against "mailto" links. As far as I can see Arelle does not yet check for these.
• Guidance 3.2.1 [removed] – Remove check for [ESEF.3.2.1.extensionTaxonomyElementNameDoesNotFollowLc3Convention]
• Guidance 3.2.5 [removed] – Remove check for [ESEF.3.2.5.abstractConceptDefinitionInExtensionTaxonomy]. Unfortunately, I cannot say for sure if this check exists, as our software does not yet allow to create extension abstract concepts
• Guidance 3.3.1 [updated] – As with the above, unfortunately I cannot check if there is validation rule already, but Arelle should raise a warning if a concept is anchored to an abstract concept
• Guidance 3.4.5 [updated] – I think "ESMA RTS on ESEF" validation must raise an error if standard labels of the ESEF taxonomy are duplicated in the extension taxonomy. E.g. "[ESEF.RTS.Annex.IV.Par.8] Extension taxonomies shall not override or replace standard labels"
• Guidance 3.4.6 [updated] – I think I remember there are checks for 3.4.6 in the ESMA RTS on ESEF validation, but I am not sure if these or some of them need to be changed fom severety level error to warning based on this new guidance?
• Guidance 4.1.1 [updated] – As far as I can see "ESMA RTS on ESEF Uncosolidated" does not disallow XHTML within ZIP files in the first hand, so probably no changes necessary.
• Guidance 4.1.3 [updated] – no implications for technical valdation

Please let me know if there are any questions.

Thanks!

internetConnectivity flag is ignored in 2022-01-31 version

In version 2021-10-17, I'm using flag internetConnectivity with value offline and I don't have any errors.
With version 2022-01-31, I'm getting the following error when I validate a report
[webCache:retrievalError] [Errno -2] Name or service not known retrieving http://www.w3.org/2001/03/xml.xsd

Arelle 1.2022.03.22 fails to open any iXBRL Taxonomy Packages

Dear all,

The latest version of arelle (2022-03-22) seems to fail opening any iXBRL Taxonomy packages.

I tried with ESEF conformance suite 2021 test G2-1-2 (but to me it seems to affect any package).

Message from Arelle is

[AttributeError] Unrecoverable error: 'list' object has no attribute 'endswith', abc-2019-12-31.xhtml - abc-2019-12-31.xhtml 
Traceback (most recent call last):
  File "Z:\Users\arelle\hfdev\arelle\ModelDocument.py", line 145, in load
  File "C:\Program Files\Arelle\plugin\validate\ESEF\__init__.py", line 106, in modelXbrlBeforeLoading
    if (modelXbrl.fileSource.selection.endswith(".xml") and
AttributeError: 'list' object has no attribute 'endswith'
[ESEF.3.1.3.missingOrInvalidTaxonomyPackage] RTS Annex III Par 3 and ESEF 3.1.3 requires an XBRL Report Package but one could not be loaded. - 
[ESEF.RTS.Art.6.a.noInlineXbrlTags] RTS on ESEF requires inline XBRL, no facts were reported. - 
not successfully loaded in 0.00 secs

Can sombody confirm this issue?

Thanks and best regards,
Sacha

Add inputs to github build workflows

The Arelle build workflows (windows, mac, and linux) hard code to use the master branch of the various plugins that are cloned into the build (EdgarRenderer, ixbrl-viewer, and xule). We need the ability to create test builds against unreleased branches of these plugins. Create workflow inputs (default to master) to select other branches from the plugin repos.

Also add inputs for python and nodejs versions (and tcl/tk for linux workflow).

Attribute error for ESEF validation

In the since the July 2022 version I get the following error message when trying to validate an ESEF reporting package:

[AttributeError] [Exception] Failed to complete request: 'ModelXbrl' object has no attribute 'validateESEFplugin' [' File "Z:\Users\arelle\devpub\arelle\CntlrCmdLine.py", line 1027, in run\n', ' File "Z:\Users\arelle\devpub\arelle\ValidateFormula.py", line 990, in validate\n', ' File "C:\InspectorFiles\Validation\Arelle\plugin\validate\ESEF\init.py", line 1105, in validateFormulaFinished\n if not (val.validateESEFplugin):\n']

This error only occurs when using the CI and using the "validate/ESEF" plugin. If I use the GUI I don't get this error. Because of the error, I think the GUI shows an ESEF error where the CI doesn't.

Table Linkbase: test case development

• Totaling variations (not in test suite)
• Roll ups testing parent-first, child-first
  • X axis 1 Breakdown, 2 breakdowns same and opposite parent-first vs child-first
  • Y axis 1 Breakdown, 2 breakdowns same and opposite parent-first vs child-first
  • Check GUI, HTML and layout outputs
• Breakdown node labeling variations (not in test suite?)
• Evaluate conformant vs production versions
  • Test suite results
  • GUI rendering, HTML output, Layout model output

Questions regarding further devops improvements

In other Python projects I'm working in, pre-commit hooks have been setup to help avoid commiting code that will not pass the CI/CD for quality reasons. Is this something you have considered doing? In that case, I'm happy to set it up.

Also, have you also considered running Black on the code?

Type Hints: FileSource.py

Add type hints to FileSource.py

Missing arg in view = ViewRelationshipSet(modelXbrl, csvfile, header)

In ViewCsvRelationshipSet.py the following line (line 12):

view = ViewRelationshipSet(modelXbrl, csvfile, header)

Should be:

view = ViewRelationshipSet(modelXbrl, csvfile, header, lang)

Table Linkbase: Review architectural changes

@hefischer table linkbase work needs to be reviewed. It is on the feature branch TLB-wip-1 and can be reviewed in PR #296.

The table linkbase branch won't be merged into master until it is certified.

• Removal of Eurofiling and other non-REC table implementations
• Implementation of structural model to conform to TLB spec and primer
• Temporary json trace output of structured model
• Adaptation of ViewWin and ViewFile modules to use conformant structural model
• Sanity check of reorganization and refactoring approach

Local mypy errors (but CI passes)

What happened?

mypy passes in CI, but when run locally it raises 4 errors in arelle/Cntlr.py:

arelle/Cntlr.py:246: error: Unused "type: ignore" comment
arelle/Cntlr.py:548: error: Item "None" of "Optional[IO[bytes]]" has no attribute "read"  [union-attr]
arelle/Cntlr.py:552: error: Item "None" of "Optional[IO[bytes]]" has no attribute "write"  [union-attr]
arelle/Cntlr.py:553: error: Item "None" of "Optional[IO[bytes]]" has no attribute "close"  [union-attr]

Expected mypy to pass without any type errors on the master branch and for CI mypy results to match local runs.

Relates to #314 #313

Documents

No response

If running from the command line, what command did you run?

mypy arelle --namespace-packages

Interface

CLI (arelleCmdLine)

Version

f033250

Download

From source (git checkout)

Operating System

macOS 12 Monterey

[ESEF] Failling test case G3-1-2/TC3_invalid

In the resource file authority-validations.json the 2019 URIs are not considered outdated, although they should be according to the 2021 conformance suite.
However I assume there's a reason why the file has been created that way, so I'm creating an issue rather than a pull request.

Upgrade Table Linkbase CI to 2015 version

What should we change and why?

@RobinetDenisAcsone pointed out that our current Table Linkbase CI is using an outdated 2014 version of the conformance suite. Upgrade to the 2015 version, available here: https://specifications.xbrl.org/release-history-table-linkbase-1.0-conformance-suite.html

Formula messages using ESEF Taxonomy 2021

While making myself familiar with the ESEF Taxonomy 2021 I noticed a difference in how the ESEF validation in Arelle (2022-04-01) handels formulae messages in comparison with the ESEF Taxonomy 2020.

When validating test RTS_Annex_II_Par_2\TC2_invalid.zip (conformance suite 2020) using "ESMA RTS on ESEF" the validation messages show:

[ESEF.RTS.Annex.II.Par.2.missingMandatoryMarkups] Mandatory elements to be marked up are missing: LegalFormOfEntity, NameOfReportingEntityOrOtherMeansOfIdentification. - abc-2019-12-31.xhtml 
[info:profileActivity] validating ESMA RTS on ESEF filing rules 0.033 secs
 - abc-2019-12-31.xhtml 
[message:man_LegalFormOfEntity] missingMandatoryMarkups: According to the Regulatory Technical Standards on European Single Electronic Format, the element "Legal form of entity" is part of the mandatory list of tags that must be applied if corresponding information is present in a report. Please make sure that you applied this tag in your report. - http://www.esma.europa.eu/taxonomy/2020-03-16/esef_cor-for.xml 20523
[message:man_NameOfReportingEntityOrOtherMeansOfIdentification] missingMandatoryMarkups: According to the Regulatory Technical Standards on European Single Electronic Format, the element "Name of reporting entity or other means of identification" is part of the mandatory list of tags that must be applied if corresponding information is present in a report. Please make sure that you applied this tag in your report. - http://www.esma.europa.eu/taxonomy/2020-03-16/esef_cor-for.xml 20491
[ESEF.2.7.1.targetXBRLDocumentWithFormulaErrors] Target XBRL document MUST be valid against the assertions specified in ESEF taxonomy, 1 with errors. - abc-2019-12-31.xhtml 
[ESEF.2.7.1.targetXBRLDocumentWithFormulaWarnings] Target XBRL document SHOULD be valid against the assertions specified in ESEF taxonomy, 1 with warnings. - abc-2019-12-31.xhtml 

In comparison, performing the same test using ESEF conformance Suite 2021 results in:

[ESEF.RTS.Annex.II.Par.2.missingMandatoryMarkups] Mandatory elements to be marked up are missing: LegalFormOfEntity, NameOfReportingEntityOrOtherMeansOfIdentification. - abc-2019-12-31.xhtml 
[info:profileActivity] validating ESMA RTS on ESEF filing rules 0.027 secs
 - abc-2019-12-31.xhtml 
[formula:assertionUnsatisfied] man_LegalFormOfEntity - http://www.esma.europa.eu/taxonomy/2021-03-24/esef_cor-for.xml 20783
[formula:assertionUnsatisfied] man_NameOfReportingEntityOrOtherMeansOfIdentification - http://www.esma.europa.eu/taxonomy/2021-03-24/esef_cor-for.xml 20751
[ESEF.2.7.1.targetXBRLDocumentWithFormulaErrors] Target XBRL document MUST be valid against the assertions specified in ESEF taxonomy, 1 with errors. - abc-2019-12-31.xhtml 
[ESEF.2.7.1.targetXBRLDocumentWithFormulaWarnings] Target XBRL document SHOULD be valid against the assertions specified in ESEF taxonomy, 1 with warnings. - abc-2019-12-31.xhtml 

No Formula Parameters or Trace options are active.

To me it seems Arelle is not retrieving the message from the formula assertion of the taxonomy, but instead the formula assertion errors, but I might be wrong and/or the terms may be technically incorrect.

I hope I did not do any wrong configuration on my end. Please do let me know if any information is missing from the issue and if you are able to reproduce this behaviour (and if it is intentional).

Thanks and happy easter.

rdflib required in version 5.0.0

During the setup, I have discovered that the library rdflib is required to be installed in the version 5.0.0 because of some namespace changes in the later versions.

Cheers.

Type Hints: ModelDocument.py

Add type hints to ModelDocument.py

How to specify OIM-CSV Dimensions has-hypercube with @xbrldt:contextElement="scenario"

Question

loadFromOIM.py creates a dimension under <segment> by default.
Is there any parameter to specify has-hypercube with @xbrldt:contextElement="scenario"?

XBRL Japan, Nobu

Repo Badges

Lots of popular open-source repos maintain a set of badges in the readme that can kind of direct people at a glance to things like required language versions, open issues, builds, versions, supported os, license, ect. Examples of some that are applicable here are:

• 
• 
• 
• 
• 
• 
• 
• 

There are bunch of cool ones you can make here showing status on build, tag, ect.

Banish re

Current imports of regex library are conditional, with fall back to re, because regex didn't exist on older Pythons.

SEC filers running from source code, using dei after 2021q4 and AuditorName AuditorLocation facts (mandatory for annual filings after 2021-12-15) are greeted with schema errors if they didn't install regex because re doesn't support pattern [\p{L}\p{N}].*

Investigate removing conditional regex imports which fallback to re (at least in XmlValidate.py). Only regex and no re anywhere?

Arelle installation via pypi

Hello, I noticed that some old Arelle versions are available on pypi (https://pypi.org/project/arelle).

Is there any plan to publish Arelle updates on that service?

Thank you!

Add type hints for files in <repo root>/arelle/

We should add type hints to the library. To make coordination easier, I suggest using this issue to mention what files we are/will be working on and tick the ones already covered. As a follow-up, we could work on the files in the sub-folders.

• init.py (empty file)
• _pkg_meta.py #315
• Cntlr.py #313
• CntlrCmdLine.py
• CntlrComServer.py
• CntlrProfiler.py
• CntlrQuickBooks.py
• CntlrWebMain.py
• CntlrWinMain.py
• CntlrWinTooltip.py
• DialogAbout.py
• DialogArcroleGroup.py
• DialogFind.py
• DialogFormulaParameters.py
• DialogLanguage.py
• DialogNewFactItem.py
• DialogOpenArchive.py
• DialogOpenTaxonomyPackage.py
• DialogPackageManager.py
• DialogPluginManager.py
• DialogRssWatch.py
• DialogURL.py
• DialogUserPassword.py
• DisclosureSystem.py
• FileSource.py #332
• FormulaConsisAsser.py #530
• FormulaEvaluator.py #531
• FunctionCustom.py
• FunctionFn.py
• FunctionIxt.py
• FunctionUtil.py
• FunctionXfi.py
• FunctionXs.py
• HashUtil.py
• HtmlUtil.py
• InstanceAspectsEvaluator.py
• LeiUtil.py
• Locale.py #341
• LocalViewer.py
• ModelDocument.py #334
• ModelDtsObject.py #534
• ModelFormulaObject.py #532
• ModelInstanceObject.py #535
• ModelManager.py
• ModelObject.py #328
• ModelObjectFactory.py #555
• ModelRelationshipSet.py
• ModelRenderingObject.py
• ModelRssItem.py
• ModelRssObject.py
• ModelTestcaseObject.py
• ModelValue.py #329
• ModelVersObject.py
• ModelVersReport.py
• ModelXbrl.py #333
• PackageManager.py
• PluginManager.py
• PrototypeDtsObject.py
• PrototypeInstanceObject.py
• PythonUtil.py
• RenderingEvaluator.py
• RenderingResolver.py
• TableStructure.py
• TkTableWrapper.py
• typing.py (contains type hints)
• UITkTable.py
• UiUtil.py
• Updater.py #308
• UrlUtil.py #325
• Validate.py
• ValidateFilingText.py
• ValidateFormula.py #533
• ValidateInfoset.py
• ValidateUtr.py
• ValidateVersReport.py
• ValidateXbrl.py #310
• ValidateXbrlCalcs.py
• ValidateXbrlDimensions.py
• ValidateXbrlDTS.py
• Version.py
• ViewFile.py
• ViewFileConcepts.py
• ViewFileDTS.py
• ViewFileFactList.py
• ViewFileFactTable.py
• ViewFileFormulae.py
• ViewFileRelationshipSet.py
• ViewFileRenderedGrid.py
• ViewFileRoleTypes.py
• ViewFileRssFeed.py
• ViewFileTests.py
• ViewUtil.py
• ViewUtilFormulae.py
• ViewWinConcepts.py
• ViewWinDiffs.py
• ViewWinDTS.py
• ViewWinFactGrid.py
• ViewWinFactList.py
• ViewWinFactTable.py
• ViewWinFormulae.py
• ViewWinGrid.py
• ViewWinList.py
• ViewWinPane.py
• ViewWinProperties.py
• ViewWinRelationshipSet.py
• ViewWinRenderedGrid.py
• ViewWinRoleTypes.py
• ViewWinRssFeed.py
• ViewWinTests.py
• ViewWinTkTable.py
• ViewWinTree.py
• ViewWinTupleGrid.py
• ViewWinVersReport.py
• ViewWinXml.py
• WatchRss.py
• WebCache.py
• XbrlConst.py #342
• XbrlUtil.py #343
• XhtmlValidate.py
• XmlUtil.py #338
• XmlValidate.py #416
• XmlValidateParticles.py
• XmlValidateSchema.py #422
• XPathContext.py #529
• XPathParser.py #555

Type Hints: ModelXbrl.py

Add type hints to ModelXbrl.py

Easy environment reporting for bugs

As suggested by @ggravlingen we should create a utility from the CLI to facilitate copy pasting environment information (or uploading a diagnostics file) for bug reports.

arelleGUI fails to download update

Version: 1.2022.06.21
OS: macOS 12.5

I check for an update, the dialog reports that version 2022-07-29 is available, and prompts me to download. I select "yes," and the dialog closes. I expect to see a download happening in the background with messages printed in the log, but it appears to silently fail--no download occurs and no error message appears anywhere.

Configure dependabot to update python dependencies

Dependabot should be configured to manage python dependency updates. With RDFLib >= 6 included in the ignore list.

Issue templates

Most of the bug reports usually require asking the reporting to provide additional information about their environment. It would be useful to have templates for common issue types such as feature requests, and bug reports that prompt the reporter to provide additional information.

ESEF extensionTaxonomyLineItemNotLinkedToAnyHypercube not detected by Arelle

An ESEF report submitted to a national authority raised an error 3.4.2.extensionTaxonomyLineItemNotLinkedToAnyHypercube when validated with the tool used by said authority:

However Arelle does not report that error:

loaded in 7.40 secs
views 3.36 secs
[ESEF.RTS.Annex.IV.Par.4.2.monetaryConceptWithoutBalance] Extension monetary concepts MUST provide balance attribute: concept econocom:CashFlowsFromUsedInOperationsAfterCostOfNetDebtAndIncomeTaxExpense. - http://www.******.com/taxonomy/2019-12-31/******-2021-12-31.xsd 748
[ESEF.RTS.Annex.II.Par.2.missingMandatoryMarkups] Mandatory elements to be marked up are missing: ExplanationOfChangeInNameOfReportingEntityOrOtherMeansOfIdentificationFromEndOfPrecedingReportingPeriod. - ******-2021-12-31-fr.xhtml 
[info:profileActivity] validating ESMA RTS on ESEF filing rules 0.402 secs - ******-2021-12-31-fr.xhtml 
Formula xpath2 grammar initialized in 0.92 secs
[formula:assertionUnsatisfied] man_ExplanationOfChangeInNameOfReportingEntityOrOtherMeansOfIdentificationFromEndOfPrecedingReportingPeriod - http://www.esma.europa.eu/taxonomy/2020-03-16/esef_cor-for.xml 20502
[message:man_ExplanationOfChangeInNameOfReportingEntityOrOtherMeansOfIdentificationFromEndOfPrecedingReportingPeriod] missingMandatoryMarkups: According to the Regulatory Technical Standards on European Single Electronic Format, the element "Explanation of change in name of reporting entity or other means of identification from end of preceding reporting period" is part of the mandatory list of tags that must be applied if corresponding information is present in a report. Please make sure that you applied this tag in your report. - http://www.esma.europa.eu/taxonomy/2020-03-16/esef_cor-for.xml 20502
[ESEF.2.7.1.targetXBRLDocumentWithFormulaWarnings] Target XBRL document SHOULD be valid against the assertions specified in ESEF taxonomy, 1 with warnings. - ******-2021-12-31-fr.xhtml 
validated in 9.73 secs

What I observed in the extension taxonomy's definition linkbase while comparing it to the one in the ESEF conformance suite test cases is that concepts are linked to a hypercube directly, using the http://xbrl.org/int/dim/arcrole/all arcrole; whereas in the conformance suite the domain-member arcrole is used.

Here is an illustration of the difference, recreated in a fictive report (attached to this issue for testing purposes)
(a variant of an ESEF test case):

After going through the XBRL Dimensions 1.0 specification I arrived to these conclusions:

• It is my understanding that dimensional validity of primary items is achieved through a (valid) link to a (valid) hypercube. (This may be an oversimplification but seems good enough in the context of this ESEF report.)

• In the ESEF conformance suite the primary items are made dimensionally valid thanks to the inheritance of the has-hypercube relationship:

• Thus making direct explicit links from concepts to the ESEF-provided “Line items not dimensionally qualified” hypercube using the all arcrole is a perfectly acceptable way to comply with guidance 3.4.2 of the ESEF reporting manual, which is about "Defining the dimensional validity of line items".

Therefore Arelle is right not to raise that 3.4.2.extensionTaxonomyLineItemNotLinkedToAnyHypercube in such a case (which makes this issue "not your problem", I guess. Or maybe my hypothesis that the issue is caused by the arcrole used is wrong). Still, could you please take a look at this and confirm that Arelle's validation of this guidance works as intended?

review:
@hefischer

Table Linkbase: Layout model output (ViewFileRenderedGrid.py)

• Group/header element structure
• Test suite validation comparison of expected vs resulting layout models

Remove Python 2 Support

Arelle contains legacy conditional code for running under both Python 2 and 3. Since Python 2 is no longer supported the Python 2 code paths can be removed.

Files with Python 2 conditional code to remove:

• apidocs/conf.py
• arelle/Cntlr.py
• arelle/PackageManager.py
• arelle/PythonUtil.py
• arelle/RenderingResolver.py
• arelle/UrlUtil.py
• arelle/ViewFile.py
• arelle/WebCache.py
• arelle/XPathParser.py
• arelle/plugin/loadFromOIM-2018.py
• arelle/plugin/saveLoadableOIM.py
• arelle/plugin/sphinx/FormulaGenerator.py
• arelle/plugin/sphinx/SphinxParser.py
• arelle/plugin/validate/USSecTagging.py
• arelle/plugin/xbrlDB/SqlDb.py

Additionally PythonUtil.py defines a number of builtins (conditional on python version) used throughout the code base. Completely remove the custom builtins and use the standard functions instead.

Table Linkbase: HTML output (ViewFileRenderedGrid.py)

• Test against prod version
• Totaling and roll-up
• Multiple breakdowns (with and without labels)
• Z axis handling

Arelle should detect duplicated facts

Some processors don't validated instances that contain duplicated facts (fact with same qname, with contexts that are equals, but not necessary the same context id).

It would be a good addition to Arelle to detect such facts.

Table Linkbase: GUI (ViewWinRenderedGrid.py)

• Totaling and roll-up rendering
• Multiple breakdown rendering
  • Without and with labels on breakdown node
• Open axis handling
• Z-axis rendering
• Table entry and edit operations
  • Open axis on X or Y
  • Open axis on Z

Use consistent CI distribution naming

What should we change and why?

The linux, windows, and macos workflows use different naming patterns for their download links. They should be consistent.

Currently:
Windows: dist
Linux: redhat distribution and ubuntu distribution
Mac: arelle-macos-12

Update to:
Windows: windows distribution
Linux: redhat distribution and ubuntu distribution
Mac: macos distribution

Incorrect qname property for ModelInlineFact and ModelInlineFootnote

The ModelInlineFact and ModelInlineFootnote instances have wrong "values" for their qname property.

In the case of footnotes, the property returns XbrlConst.qnLinkFootnote which is not correct for inline XBRL.

For facts the property returns the concept's QName.

Failure to load plugin causes GUI to crash

If the GUI has a plugin enabled that can't be found it will crash at startup with the following error:

Exception at plug-in method freshenModuleInfos: [Errno 2] No such file or directory: '/arelle/arelle/plugin/'

This can happen if you have used one of the pre-built distributions of arelle to enable a plugin that's not part of the core arelle repo (EdgarRenderer) and then later try to launch the arelle GUI from within the repo (python arelleGUI.pyw).

If the arelleCmdLine can't find a plugin it will log an error, but continue processing. The GUI should gracefully handle missing plugins in the same manner.

Configure dependencies for pypi package

You should be able to install arelle from a pypi package into a new project and use it without manually installing required dependencies.

The dependencies for the pypi package (install_requires in setup.py) are incomplete. It only contains lxml, isodate, and openpyxl. The list needs to be updated to include all necessary dependencies (including plugins that are directly committed to the arelle repo).

Additionally, dependencies in setup.py should specify support version ranges (latest working major version).

Calling the controller in python directly from the file wants a nonexistent directory

Since this is the only instructions for the Python API https://arelle.org/arelle/documentation/api/ I've run into issues in creating an instance of the controller (which appears to be required).

I get a strange error when trying to import and instantiate the class directly

from arelle import Cntlr
ctlr = Cntlr.Cntlr()

PermissionError: [Errno 13] Permission denied: '/home/nagat'

It appears to be trying to point to a directory from one of the devs. I cant find the reference in Cntlr.py . Could it be leaking from whomever created the build? There aren't any arguments to specify any directories, so I am not sure what to do.

Validation slow for typed dimension tables in Solvency2

When validating XBRL files containing dynamic tables like in Solvency2 or CRDIV, the validation takes significantly more time than other validation tools. I created a test to reproduce this issue using the S.06.02.01 tables in Solvency2:

I added two tests. One with 10 lines and one with 1000 lines. The file containing 10 lines is validated relatively quickly, but the one containing 1000 lines will take a significant amount of time (after 30 minutes I stopped the process). Meanwhile, a validation tool like Altova validates that file in a couple of seconds.

2.6.0.zip

I have used version 1.2022.05.27 64-bit for these tests.

Unknown color name "SystemHighlight" error on opening a file

Exception details:

Exception preparing tree view of tests: unknown color name "SystemHighlight", at [' File "/home/mike/Arelle/arelle/CntlrWinMain.py", line 522, in showLoadedXbrl\n ViewWinDTS.viewDTS(modelXbrl, self.tabWinTopLeft)\n', ' File "arelle/ViewWinDTS.py", line 13, in viewDTS\n view = ViewDTS(modelXbrl, tabWin)\n', ' File "arelle/ViewWinDTS.py", line 23, in init\n super().init(modelXbrl, tabWin, "DTS", True)\n', ' File "arelle/ViewWinTree.py", line 25, in init\n highlightColor = "#%04x%04x%04x" % self.treeView.winfo_rgb("SystemHighlight")\n', ' File "/usr/lib/python3.2/tkinter/init.py", line 781, in winfo_rgb\n self.tk.call('winfo', 'rgb', self._w, color))\n']

I tried Arelle with a system-provided Python3:

$ aptitude show python3-tk
Package: python3-tk
State: installed
Automatically installed: no
Version: 3.2-1ubuntu1
Priority: optional
Section: python
Maintainer: Ubuntu Developers [email protected]
Uncompressed Size: 160 k
Depends: python3 (>= 3.1), python3 (< 3.3), blt (>= 2.4z-4.1), libc6 (>= 2.4), libx11-6, tcl8.5
(>= 8.5.0), tk8.5 (>= 8.5.0)

Extract value from roleRef in ESEF report

Hi,

I'm running version 0.9.91 and have successfully been able to extract facts from an annual report published under the ESEF-format. It would be very helpful to add further context to the facts by including a value found under roleRef. The excerpt below is found in a file named xyz-2021-12-31_cal.xml and for each fact, I'd like to know if it belongs to the role ProfitOrLoss, StatementOfFinancialPosition or StatementOfCashFlows. Can this be achieved?

<roleRef roleURI="http://atlascopco.com/roles/ProfitOrLoss" xlink:href="atla-2021-12-31.xsd#ProfitOrLoss" xlink:type="simple" xmlns:xlink="http://www.w3.org/1999/xlink" />
<roleRef roleURI="http://atlascopco.com/roles/StatementOfFinancialPosition" xlink:href="atla-2021-12-31.xsd#StatementOfFinancialPosition" xlink:type="simple" xmlns:xlink="http://www.w3.org/1999/xlink" />
<roleRef roleURI="http://atlascopco.com/roles/StatementOfCashFlows" xlink:href="atla-2021-12-31.xsd#StatementOfCashFlows" xlink:type="simple" xmlns:xlink="http://www.w3.org/1999/xlink" />

CI mac distribution "is damaged"

The Arelle.app produced from the CI workflow can't be run without first removing extended attributes.

xattr -dr com.apple.quarantine /Applications/Arelle.app