Product roadmap or wishlist (from source maintainers)? about capirca

[vklimovs]

We don't have formal plan, but really our plan is pretty simple - deprecate internal version only have this open source version. What we have internally that open source version does not have is pretty decent tests for all generators, so this you do not need to develop.

Once the sync is complete, I'd have no problem accepting any/all reasonable patches.

[jzohrab]

Following up ... I would like to know so that I can contribute effectively to the project. I would prefer to not fork!

[ankenyr]

Hi jz,
Sorry for the lack of response. I don't think we have anything put together yet as a road map or a wish list but I think we could put something together. I created a wiki page with an initial entry that I can think of located https://github.com/google/capirca/wiki/Capirca-Wishlist
I will sync up with the other team members. There are some bugs I know of but most of them have to do with internal systems that don't make sense to include in Capirca.

I agree us duplicating work is not great so this wishlist is a great idea, thank you for bringing it up.

[jzohrab]

Hello @ankenyr - np and thank you for taking the time to respond to a random internet guy. I may need to fork as our current requirements aren't exactly met by Capirca - but maybe you and others would find our plans for additional features useful:

• we need to generate the policies for several devices. It makes sense for us to have a separate input source for devices, interfaces, directions and policies that we'd like to apply (e.g., "apply policy P to device a.b.c's f0/1.20 IN"), and the device a.b.c's platform and IP etc would be pulled in from a separate data source/CMDB.
• since we'll be feeding the policy data from different sources, it may make sense to pull some of the validation out of the lexx/yacc parser and beef up the model layer, and perhaps pull the parsing out to a separate module to refine the model layer
• add unit tests/characterization tests as needed

If any of this sounds like it would be useful for you, please let me know. I feel I'll have to make some non-trivial changes to parts of the code to support our needs, but would really like for these to be useful for others, I think the changes will make sense.

If there is a document or mailing list that I can contribute to, I'm very happy to do so.

Once we have our POC done (I can share that if you're interested, either before or afterwards via doc or skype etc), I'll probably want to start refactoring places in the code if I feel the clarity could be improved substantially and if it helps our team. I'll only refactor things once tests are in place.

I'd welcome a chance to contribute, and thanks very much for making your work public.

Regards, jz

[jzohrab]

OK, thanks. To make progress I'm forking for now and pushing the changes up to GitHub, https://github.com/jzohrab/capirca/commits/development. There are some commits there that may be useful, I can see about contributing them back individually when you've made your release. Or I could just PR the entire development branch.

There were only a few API-level additions, otherwise everything is just adding sanity checks/model-level checks.

[jzohrab]

Just mentioned in issue #58 - I'm still working on my dev branch at https://github.com/jzohrab/capirca/tree/development. The changes are starting to accumulate, but I'm happy with how things are going and feel they make sense. If there is a way to collaborate with the Google team I would be very happy to do so.

I'll stop posting here, let me know if you'd like to go over my work. Thanks and regards, jz

[jzohrab]

Copying over discussion from #58:

Note from @ankenyr on the work done in the internal Google code: "Here are some diffstats for some of the more critical files from my point of view:"

$ diff aclgen.py tools/aclgen.py | diffstat 
unknown | 725 +++++++++++++++++++++++++++++++++++++++++++++-------------------
1 file changed, 521 insertions(+), 204 deletions(-)

$ diff lib/cisco.py lib/cisco.py | diffstat 
unknown | 242 ++++++++++++++++++++++++++++++++++++----------------------------
1 file changed, 139 insertions(+), 103 deletions(-)

$ diff lib/aclgenerator.py lib/aclgenerator.py | diffstat 
unknown | 85 ++++++++++++++++++++++++++++++----------------------------------
1 file changed, 41 insertions(+), 44 deletions(-)

$ diff lib/juniper.py lib/juniper.py | diffstat 
unknown | 363 +++++++++++++++++++++++++++++++++++++++++++++-------------------
1 file changed, 259 insertions(+), 104 deletions(-)

$ diff lib/iptables.py lib/iptables.py | diffstat 
unknown | 25 ++++++-------------------
1 file changed, 6 insertions(+), 19 deletions(-)

$ diff lib/nacaddr.py lib/nacaddr.py | diffstat 
unknown | 53 ++++++++++++++++++++++++-----------------------------
1 file changed, 24 insertions(+), 29 deletions(-)

$ diff lib/junipersrx.py lib/junipersrx.py | diffstat 
unknown | 557 +++++++++++++++++++++++++++++++++++++++++++++++++++-------------
1 file changed, 452 insertions(+), 105 deletions(-)

$ diff lib/naming.py lib/naming.py | diffstat 
unknown | 62 +++++++++++++++++++++++++++++++++-----------------------------
1 file changed, 33 insertions(+), 29 deletions(-)

$ diff lib/policy.py lib/policy.py | diffstat 
unknown | 444 +++++++++++++++++++++++++++++++++++++++++++++++++++-------------
1 file changed, 355 insertions(+), 89 deletions(-)

[jzohrab]

My development branch changes

Below is a summary of my development branch changes as at 2016 May, from where my development branch diverged from Google master:

$ git log origin/master..development --oneline --reverse | head -n 1
924df32 Improve test error output.   # Initial point of divergence

$ git diff 924df32^..HEAD --numstat | grep -v test   # Ignore tests

# add   del     file
2   0   .gitignore
10  0   Makefile
0   0   __init__.py
270 132 aclgen.py
460 0   doc/policy_format.md
0   310 doc/policy_format.txt
131 0   doc/policy_format_yml.md
2   2   lib/iptables.py
54  993 lib/policy.py
1024    0   lib/policyparser.py
531 0   lib/yamlpolicyparser.py
1   1   policies/sample_multitarget.pol
2   2   setup.py

Each commit has been documented in the commit messages. I've started to do pull requests into my development branch for clarity, hoping that it would help the discussion of the work and increase the chances of me merging back into google/master. PRs are here: https://github.com/jzohrab/capirca/pulls?q=is%3Apr+is%3Aclosed:

pull/1: Separate parsing from model

• This PR splits policy.py into policyparser.py and policy.py. This will allow for additional parsers with different grammars (e.g, yaml) and requirements to be introduced, and clarifies the model layer.

This PR accounts for the large changes in policy.py, and the addition of the policyparser.py

pull/2: Introduce AclGen object

• Preparatory work for introducing new parsers.
• ACL generation needs to be able to work with different data parsers. This work will let us substitute in a different parser (to support different grammars, different input file requirements, etc). Some methods from aclgen.py are left as module-level methods (which delegate to the AclGen object) for backwards compatibility.

This looks like it will conflict with your changes in tools/aclgen.py.

pull/3: Yml parser

• Add yml parser, with tests. Reasons for doing so are documented in https://github.com/jzohrab/capirca/blob/development/doc/policy_format_yml.md.

This PR also added a very small breaking change for specifying policy file includes.

Future work

• rework naming.py. This code is quite messy and doesn't have any real test coverage. Most of the parsing can probably be removed and replaced with a YAML format parser, in which case I'd leave the code as-is and create a new YAML parser like I did with policies.
• remove duplicating in yamlpolicyparser and policyparser. The former started as a copy of the latter for expediency, and the common parts could be pulled out.
• refactoring. There are many places that could use some test coverage and refactoring - eg, simplifying some of the output, moving error class definitions to a separate module for clarity, etc. I've not done much of this work as it would increase the chance for conflict on code merge.
• documentation, examples, etc. I started this in the doc folder.

[kkirsche]

Minor typo I believe on the wiki's page for this:
https://github.com/google/capirca/wiki/Capirca-Wishlist-and-Roadmap

I believe "6. Use Tavis for testing." should be "6. Use Travis for testing." The wiki seems to be locked to edits so had to mention it here

[ankenyr]

Thanks for catch kkirsche!

[ankenyr]

I am going to close this as I think the discussion here is no longer needed. We made a simple road map. If others are doing significant work such as making a new generator or have something they want to add to the wishlist please feel free to suggest an edit.