impact / impact Goto Github PK

While this has already been completed for the refresh operations, @dietmarw pointed out that it is still present for the install action.

It should be possible, as with pip, to install something even if it hasn't been indexed by impact, e.g.,

impact install -e git+https://github.com/xogeny/Sensors.git#/path/to/library

The main application for this, that I see, is the ability to load private libraries on GitHub, but I suspect there will be others. Note, this only works for the initial library to be downloaded. Any other libraries have to be indexed (or else impact can't know about them).

If other private libraries are required, then I see two possible options. First, the user could create a private index file (so that impact could find and load additional dependencies). Second, the user could list additional libraries in the install command as a way to preemptively load dependencies that are private.

Some users prefer to run impact with Python 3 installed. So impact should not fail for them. At the same time we should stay Python 2 compatible, which should be possible.

So far people can use impact.py to search and install. But it would also be nice to list the available repos. I think a impact.py list option should be added. The output could then look like:

$ impact.py list
PackageA <URL>
PackageB <URL>

and with

$ impact.py list -v
PackageA "description" <URL>

The should point to the "homepage" value from the GitHub API which we also need into the refresh call then.

The plan is to have to execute impact without the python extension. This is also how it is described in the paper.

When trying to deploy impact.py on the server I noticed that any $HOMEDIR based config file location is not good. On the server one would like to run the script with a $HOMEDIR -less user or simply a user whos $HOMEDIR is not suitable (e.g., www-data has /var/www as home dir where we really don't want to put config files)

How about a -c --config option to define the location/name of the config file. Simply keep the current default standard location as fall back in case no -c option is given.

Instead of doing:

impact.py install LibName

It would be good to also allow a syntax like this:

impact.py install git://github.com/LibName#X.Y.Z

I think PIP supports a syntax like this.

A link to the Modelica 2014 paper could be added to the ReadMe.md
http://dx.doi.org/10.3384/ECP14096543

Currently impact.py is not able to install MSL 3.2.1 because the tag name of that is v3.2.1+build.2. The +build.2 bit cause the Modelica root not to be found. But simply dropping that bit when looking for the Modelica root causes problems as soon as there are several builds available.

I suggest the following solution:

1. when looking for the Modelica root within a tagged version, simply also look at <package> <version number without trailing "+.*" or "-.*">. After all those are not yet legal Modelica identifiers anyway.
2. Install into <package> <version number without trailing "+.*" or "-.*">

Example:
impact.py install Modelica 3.2.1+build.2
will install into "$MODELICAPATH/Modelica 3.2.1"

I just noticed that currently we can't actually specify the to-be-installed version/tag, see #30.

Currently the package matching a search term will be displayed in a random sequence. We should sort this alphanumerical.

In general, I think thematically it makes sense to have a visual of a red ball colliding with something. One option would be a Newton's Cradle (turns out there is already swag available for it:

https://www.zazzle.com/rlv/newtons_cradle_silver_balls_concept_mugs-r2209a69b1f0644c19361ea5f3b4ac845_x7jgr_8byvr_512.jpg

Another idea would be to have a red ball really smashing into something. That's harder to draw though. :-)

It would be cool to get a few items of swag to hand out at the Modelica Conference. :-)

Being able to supply an output, e.g.,
-o --output
argument makes it easier to deploy the refresh script on the server.

I noticed that the SPOT library returns no available version despite https://github.com/modelica-3rdparty/SPOT returning:

[
  {
    "name": "v0.706",
    "zipball_url": "https://api.github.com/repos/modelica-3rdparty/SPOT/zipball/v0.706",
    "tarball_url": "https://api.github.com/repos/modelica-3rdparty/SPOT/tarball/v0.706",
    "commit": {
      "sha": "cc277376fb50cc407e15d3493f6d71ad38157350",
      "url": "https://api.github.com/repos/modelica-3rdparty/SPOT/commits/cc277376fb50cc407e15d3493f6d71ad38157350"
    }
  }
]

So something is not right with the non-semver to semver padding.

It occurs to me that creating private repository indices will require credentials on
a per repository basis. We don't really have a mechanism to do that right now.

One possibility would be to incorporate it into the URL listed in the index somehow.

Because the MSL is stored in a repo named ModelicaStandardLibrary, we have a problem with the scripts.

I'm looking at some hacks for now. But the simple solution is to rename the repository to follow the conventions that we expect everyone else to follow.

I think it would be useful to add a command line option to impact that
tells the user where to put a user specification configuration file. The logic is so convoluted that I think it is easier to just say "put it here" than explaining how the
user can "compute" the expected location.

Looking at the generated json file, tags containing a "v" at a later than the first position cause some unexpected beaviour.

Example from the log file:

Tag: v1.5-dev
  Semantic version info: 1.5.0-dev
    Path: .
    Dependencies: []
  Also storing under version: 1.5-de

That should be stored as 1.5-dev.

Currently, we do not extract dependencies from libraries. This would be really useful because when someone requested a given package, we could automatically load all dependencies (and detect conflicts in those dependencies).

There are really two parts here. The first is to extract dependency information. This shouldn't be too hard to do (assuming we can identify the correct package.mo file to parse for the info). The second part is identifying the proper version of all libraries when loading multiple libraries.

In the future, I can see applications where we would want to optimize things to know whether we need to download/upgrade things. In this case, we should include (optionally?) the hash associated with a given tag.

This gives some sense of community interest.

Currently, dependencies are listed as essential a tuple of the form:

["<LibraryName>", "<LibraryVersion>"]

I think it would be much clearer (certainly in the schema) if this was:

{"name": "<LibraryName>", "version": "<LibraryVersion>"}

Changing the schema should be simple. But I have not yet investigated the
impact in changing the code. It would mean a change to where the index
information is created and then where it is read.

I'm going to fix this in a minute.

I noticed that the -d dryrun option does actually pull down all the libraries but does not install them. Wouldn't it be sufficient to just check if the download link works (200 OK) and then skip the actual download? It would make the test much quicker.

Currently the versions as seen by impact include the +build.x part not just for listing the available versions (which is fine) but also for installation.

The problems are:

1. When a user does impact install Modelica#3.2.1 nothing is getting installed since impact expects the complete version number including the +build.x part.
2. Dependencies in library only state the version number without any build.

Solution:
When a users does impact install Modelica#3.2.1 simply install the latest available build and the same also when a version number is given as dependency. Still keep the functionality to explicitly state a to be installed build number.

The options -d and -v are described in the paper, they might be added to the ReadMe, too. Others might also be useful.

Whilst working on the impact-paper I noticed that impact currently uses the -d switch for:

• dry-run in combination with impact install
• display the description string in combination with impact search

I really like to have consistent switch names at least within the same tool that don't change the meaning. Now either we change the dry-run to something else or (what I would prefer) we combine lock the output of description string to the -v switch. So that we get then displayed the description string and the available versions at once.

It would be good if impact optionally could clone in addition to unzipping dependencies. I'm not sure how we should indicate when to clone and when to unzip, so we should think about that.

This will be necessary to support interoperability with other clients (i.e. tool
vendor implementations).

I just tried to use impact on Windows 7 64 bit, but I failed to do so.
I installed impact using

• easy_install impact under Python 2.7 and
• pip install impact under Python 3.4 (Python 3.4 comes with pip included).

Both times something was installed under C:\Python27(or34)\Lib\site-packages\impact-xyz. But I don't know how to use it. There is no .exe that could be run from the Windows command prompt, only Python scripts. But when I try to run the commands shown in your paper from a Python shell I just get either

• SyntaxError: invalid syntax or
• NameError: name 'impact' is not defined.

I am pretty sure I am missing something obvious, but I don't see what. Could you help me out?

We should be able to define an installation directory so that one can point to the $MODELICAPATH or whatever installation location one prefers.

Shoud be an config argument of "install" and an entry in impactrc.

What about -t --target and

target_dir=

Since this repo is now also public I wonder if I should import the paper repo (including the history) into this repo under, let's say, docs/modelica2014?

There should be an option to give a user and repo name and have the same code that runs refresh run for just that repo. This will allow developers to debug why there repos don't get scanned.

It turns out that accessing SSL resources on modelica.org programmatically is a problem on Windows. Apparently, there is some known issue with using SSL on Windows.

That link includes a solution which I incorporated into load.py, but haven't tested sufficiently.

For the paper, I'm using the term "index" or "indices" to refer to the data that is
generated by impact. As such, we need to change the names of some functions
and some user settings to reflect this.

@casella pointed out some valid points on what if a to be installed Modelica package needs to be tweaked in order to work with the tool. In this case a stock-source library form the official repository will fail as it might lack certain compiled parts.

I propose to make impact aware to already installed Modelica packages in the target directory and to not overwrite them by default if the same or later (build) version is present. There should be a -f | --force switch available installing such Modelica packages anyway.

I think we could probably do this technically. But should we encourage this?

For example, that git tags contain a version but unless the two libraries are in lock step, how do handle versioning.

Furthermore, you have the problem of identifying all the libraries in the repo.

Personally, I think we should encourage people to break these things out. If we don't, impact will have a huge pile of requirements in order to support all this complexity.

Comments?

Many libraries still depend on "modelica-deprecated" libraries. Question is, should we also make those available or should we simply say that those >10 year old libraries ought to be updated in order to be used properly with impact.py?

When doing a pip install impact only impact.py gets installed as executable. But also the refresh.py should be made available.
This again triggers the question if refresh.py should probably be integrated as option to impact.py rather than a sole script. On the server I rather have one impact.py application that I can run with options than a refresh script that I either have to locate or when installed as /usr/local/bin/refresh.py is not really obvious to belong to impact.py

It would be much cleaner to use
$XDG_CONFIG_HOME (defaults to $HOME/.config)

as place to store the impact configuration. the total path would then be:

$HOME/.config/impact/impactrc

This is probably also easier to port to Windows using the %APPDATA% (or what it is called again) env variable.

The API only returns zipball and tarball links as as API calls. This means that if users don't have a token configured they will pretty quick hit the API restrictions.

Solution would be to find the pattern of the zipball links and replace the API ones with normal web links.

When working on #65 I ran pyflake3 to check if the code is correct python 3.
Pyflake3 (and pyflake2) reported also a serious of unused imports which I could fix in connection with #65.

pyflakes3 *.py
load.py:23: local variable 'e' is assigned to but never used
semver.py:63: undefined name 'cmp'

now the issue in load.py is not critical but I don't know if we should do something about the error in semver.py

When doing impact search <search term> the case of the <search term> should not matter.

Currently, the zipball archive from GitHub is just extracted directly in the current directory. But this isn't really the package itself. Instead, the package is inside the directory created during extraction.

We should identify the correct directory (within the extracted directory) and lift it into the current directory (and then discard all the extra junk). In some cases, the directory already has a version number in the name, in other cases it doesn't.

Given that we know the specific version being installed, if the directory (or file name?) doesn't have an explicit version number, we should add one. This will make it very clear what version we have. This could be very useful when adding additional dependencies after some initial ones because it will allow us to detect potential conflicts.

Most libraries available in GitHub depend on the Modelica Standard Library. However, most versions of the MSL (and in particular, 3.2 which is the most heavily depended on one) do not conform to semantic versioning. So if we want to make impact immediately useful, we need to find a way to support these legacy ones but try and enforce stricter adherence in the future.

Whilst working on the Python 3 conformity in #65 I had to add another decode of the dependency parsing and ran into the problem that some libraries still use latin1 encoding. In that case a simple warning is created. If however the modelica files are neither UTF8 (as they should be) nor latin1 (as we still allow here) the dependency check aborts, writes out an error message and returns null for the dependency field.

There might be a better way dealing with this, hence this ticket.

Note: Currently none of the modelica-3rdparty libraries have an encoding different from latin1 or UTF8

A couple of libraries (e.g., Modelica_StateGraph2), throw currently an ugly Python error because impact finds several dependencies and self-dependencies. Now rather than exiting we should probably simply report those illegal nested "uses" annotations and only use the top-level one.

We should technically check the version annotation to make sure it matches the tag we are using to infer version number.

when doing impact.py search Modelica I get also packages that do not contain "Modelica" in their name. This is not the expected behaviour as there is the option -d to include the description (for the search or only for the display???)

I would like to be able to do impact.py install Modelica 3.1 or more sane impact.py install "Modelica 3.1"

Bower and NPM both (can) place a file in the root directory of a project
listing the dependencies. This is nice because it means that the stuff these
package managers download doesn't need to be version controlled. Instead, the
presence of these json files is sufficient to reconstruct all dependencies.

To support this, it would probably be a good idea to implement an init command
to create the initial JSON dependency file. This should probably also walk the user
through some questions to populate this file.

If this is supported, then the equivalent of the --save-dev or -s flag would need
to be implemented to rewrite the dependencies file whenever a new library is
installed.

Finally, it is worth noting that an approach like this could completely obviate the
need for version information in Modelica files. We are probably too far down that
road to switch to such a different approach. This means that this feature would
probably amount to a completely redundant and potentially inconsistent way
of storing version information about libraries.