The problem seems to be with beacon servers.
Here's the output of logs/bs1-15-1.log (truncated):
[WARNING] (BS shared pcbs) AD: Data too short for parsing, len: 20
[CRITICAL] (BS shared pcbs) Exception in handle_shared_pcbs thread:
[CRITICAL] (BS shared pcbs) Traceback (most recent call last):
[CRITICAL] (BS shared pcbs) File "/home/scion/lib/thread.py", line 51, in wrapper
[CRITICAL] (BS shared pcbs) return f(*args, **kwargs)
[CRITICAL] (BS shared pcbs) File "beacon_server.py", line 608, in handle_shared_pcbs
[CRITICAL] (BS shared pcbs) count = self._read_cached_entries()
[CRITICAL] (BS shared pcbs) File "beacon_server.py", line 635, in _read_cached_entries
[CRITICAL] (BS shared pcbs) count = self._process_cached_pcbs(new, timed_desc=desc)
[CRITICAL] (BS shared pcbs) File "/home/scion/lib/util.py", line 214, in wrapper
[CRITICAL] (BS shared pcbs) ret = f(*args, **kwargs)
[CRITICAL] (BS shared pcbs) File "beacon_server.py", line 664, in _process_cached_pcbs
[CRITICAL] (BS shared pcbs) pcbs.append(PathSegment(raw=raw))
[CRITICAL] (BS shared pcbs) File "/home/scion/lib/packet/pcb.py", line 286, in __init__
[CRITICAL] (BS shared pcbs) self.parse(raw)
[CRITICAL] (BS shared pcbs) File "/home/scion/lib/packet/pcb.py", line 309, in parse
[CRITICAL] (BS shared pcbs) self.add_ad(ad_marking)
[CRITICAL] (BS shared pcbs) File "/home/scion/lib/packet/pcb.py", line 327, in add_ad
[CRITICAL] (BS shared pcbs) if ad_marking.pcbm.hof.exp_time < self.min_exp_time:
[CRITICAL] (BS shared pcbs) AttributeError: 'NoneType' object has no attribute 'hof'
[CRITICAL] (BS shared pcbs)
[INFO] (MainThread) Received SIGTERM
[INFO] (MainThread) Exiting

As discussed with Lorenzo, BS/CS store TRCs and certificate chains in memory.

Type Field of Info Opaque is 7 MSB bits.
But, TRC_OF has 8 bits.

https://github.com/netsec-ethz/scion/blob/master/lib/packet/opaque_field.py#L38
TRC_OF = 0b11111111

Is it right?

Why is PT.PATH_MGMT specified twice?
https://github.com/netsec-ethz/scion/blob/master/infrastructure/router.py#L350

At the moment, we don't have any means to check bit errors at the SCION packet level, which could cause any kind of troubles. For now we avoid this problem by having SCION packet actually tunnelled through IPv4, which does the bit error detection for us, however, if we ever replace IP with SCION, we have to have some kind of CRC in place to catch bit errors.

I tried to run Docker (as described in the README file) in a clean Ubuntu VM, but I got the following error:

./docker.sh build
[...]
chfn: PAM: System error
adduser: `/usr/bin/chfn -f ZooKeeper zookeeper' returned error code 1. Exiting.
dpkg: error processing package zookeeper (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of zookeeperd:
 zookeeperd depends on zookeeper (= 3.4.5+dfsg-1); however:
  Package zookeeper is not configured yet.

dpkg: error processing package zookeeperd (--configure):
 dependency problems - leaving unconfigured    
[...]
Errors were encountered while processing:
zookeeper
zookeeperd
E: Sub-process /usr/bin/dpkg returned an error code (1)
2015/07/24 17:14:04 The command [/bin/sh -c bash -c 'DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y $(< docker/pkgs_preinstall.txt)'] returned a non-zero code: 100

A lot of the packet library prints a warning when it encounters a bad packet, which then causes code that relies on it to die in unexpected ways. A classic example is this:

https://gist.github.com/kormat/598e77bb164483d217e4

We need to do an error-handling audit to fix these cases with proper exceptions.

process_trc_request:
https://gist.github.com/kormat/f079e32a3a32de081e83

I can reproduce it in docker by repeating this 1-2 times:

./scion run
# Wait maybe 20s
./scion stop
./scion topology

Currently we have mixed flags and types. @LorenzoBaesso I cannot find your comment with a proposition for that, if you can find that please paste it here.

Right now it's hard to follow them easily as one as to jump up and town within the file.

Having continuous integration for SCION would make it much simpler and safer to ensure that new PRs won't break existing code/tests.

It seems to be unimportant, but sometimes (once per ~50 executions) it makes end2end test failing. In most such cases one router does not start (it even does not create a log file). Also this bug is quite difficult to replicate.

Recap on path servers design:

Path Servers (PSes) that receive request/registration are selected via anycast

Core

Down-paths registrations:

1. One master PS is elected (through ZK).
2. A registration is received by a PS, which:
   • registers the path
   • forwards the registration to the master, that registers the path
     as well
   • forwards the registration to all other ASes in the ISD core,
     that accept the registration and replicate it in the same way
     (i.e., masters have all down paths from the ISD)

Down-path request:

1. Requested destination is within a local ISD:
   • a PS is queried and he responds with a path if he has one
   • if path is not found, the master is asked about a path, and if master has one, the path is returned to the queried PS
2. Requested destination in within a remote ISD:
   • queried PS returns path is he has one, otherwise:
   • queried PS contacts any remote PS from the remote ISD. The remote PS
     returns the path, that is cached by the requesting PS and
     returned to the initial requester

Core-path registration/requesting:

1. registration is sent to a PS, that caches and replicates as follows:
   • if core-path is to AS from local ISD, then this path is fully replicated (i.e., through ZK)
   • if core-path was initiated by AS outside local ISD, then this path is sent to master PS
2. request is handled by any PS
   • returns a path between requested ASes if there is any cached, otherwise:
   • if requested path is to remote ISD, then query a master about that (and cache returned result)

When master fails:

1. New master PS is elected.
2. After election all PSes send their cached local down-paths (i.e.,
   down-paths from local ISD) to the master, and core-paths to remote ISDs.
   • due to efficiency reasons, only single path for (src, dst) pair is sent
3. Core-paths originated within local ISD are known to new master through ZK cache

Non-core:

Up-path registration:

1. Registration is received by a PS and replicated via ZK.

Up-path request:

1. Any PS handles it.

Remote paths are cached only by PSes that obtained it

PSz: Stephen has proposed a caching proxy (for paths and DNS queries).
It can provide nice features (like end-hosts without topology files)
but I would not consider this as something urgent.

Revocations are replicated through ZK

Each BS maintains its own interface revocation state independently, so as soon as an interface is revoked, other BSes are immediately out of step.

(#129)

@shitz is there any reason why revocation token for egress interface is present in every {PCB/Peer}Marking? One per AD block (i.e., ADMarking) should be enough.

@LorenzoBaesso I looked at path-store.py, it seems like incomplete and untested.
E.g. get_candidates() is executed, however ... it is not implemented.
store_selection() or remove_segments() is not even executed.
Does PathStore instance free memory ?

I'm going to resize (ISD, AD) pair, as current 8 bytes long AD is quite large.
I propose to use 4 bytes for both, as 4 would fit packet header well for IPv{4,6}, but I'm not sure how we should partition that.
Propositions:

• 1B for ISD, 3B for AD, (isn't 256 as a total number of ISDs too restrictive?)
• 2B, 2B (intuitively, there should be more ADs per ISD, than ISDs at all)
• 12bits, 20 bits (probably this is a good trade-off, however do we want to deal with bitwise operations for addresses?)

In general, I don't have any strong arguments for/against given layout, so I would like to hear what do you think...

Currently, there is no method exposed through lib.zookeeper to delete a PCB from the shared path. Only an expire_shared_items is provided that, in its current implementation, doesn't make much sense.

We should decouple expiration times from the ZK part of the code entirely, since path_store already handles removing expired PCBs. To be more precise, we need X things:

1. Expose a method to delete a PCB with a given identifier through the lib.zookeeper API.
2. Decouple expiration times from ZK. In particular this means getting rid of expire_shared_items.
3. Enhance PathStore to take a delegate object that gets called when certain things happen. For now this would mainly involve the deletion of a PCB from the store.

https://gist.github.com/kormat/7ba022a0c561b1e6fd27 is a backtrace.

I reproduced this in docker by running this a few times:

./scion.sh stop; ./scion.sh topology; rm logs/*; ./scion.sh run

In order to remove Beacon Servers as SPOFs, we're moving to a distributed architecture. The current proposal is based on Zookeeper, with the Kazoo python client library.

All Beacon Servers (BS) in an AD connect to a Zookeeper (ZK) cluster. They register as part of a Party; this is used to create multiple DNS entries for the Beacon service. The BSes try to grab the master lock, whichever BS succeeds is the master until failure/shutdown.

When an Edge Router (ER) receives a new PCB, it connects to any BS in the DNS record. If that connection times out, the ER tries the next record, and so on. The receiving BS drops the new PCB into the shared ZK "incoming" directory. All BSes have watches set on this directory, and load any new PCBs into their PathStores.

Once per propagation period, the master BS will select k PCBs from Pathstore (using freshness criteria to make sure nothing stale is selected) and propagate them. It will also move all PCBS from the ZK "incoming" directory to the "recent" directory, clearing the "recent" dir beforehand. Any newly-started BS will load all PCBs from both "incoming" and "recent" directories to prime their PathStores.

It fails in with the following:
Setting up zookeeper (3.4.5+dfsg-1) ...
chfn: PAM: System error
adduser: /usr/bin/chfn -f ZooKeeper zookeeper' returned error code 1. Exiting.
dpkg: error processing package zookeeper (--configure):
subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of zookeeperd:
zookeeperd depends on zookeeper (= 3.4.5+dfsg-1); however:
Package zookeeper is not configured yet.

dpkg: error processing package zookeeperd (--configure):
dependency problems - leaving unconfigured
Processing triggers for libc-bin (2.19-0ubuntu6.6) ...
Processing triggers for ureadahead (0.100.0-16) ...
Errors were encountered while processing:
zookeeper
zookeeperd
E: Sub-process /usr/bin/dpkg returned an error code (1)

Here is the full output: https://gist.github.com/pratyakshs/f498206cf6454bd5a07a

Host OS:

$ uname -a
Linux pratyaksh 3.16.0-38-generic # 52~14.04.1-Ubuntu SMP Fri May 8 09:43:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.2 LTS
Release: 14.04
Codename: trusty

Docker version:

$ docker --version
Docker version 1.0.1, build 990021a

This is on current master:

$ PYTHONPATH=../../ python3 ./end2end_test.py
You can specify src and dst by giving 'sISD,sAD dISD,dAD' as the arguments. E.g.:
# python3 end2end_test.py 1,19 2,26

Testing: (2, 26) -> (2, 25)
Bound 127.2.26.254:30040
Bound 127.1.19.254:30040
Local API 127.255.255.254:3333
Sending PATH request for (2, 25) in 2 seconds
Sending path request to local API.
API: path request from ('127.0.0.1', 5005).
Traceback (most recent call last):
  File "./end2end_test.py", line 191, in <module>
    TestSCIONDaemon().test(sources, destinations)
  File "./end2end_test.py", line 170, in test
    self.assertTrue(ping_received)
  File "/usr/lib/python3.4/unittest/case.py", line 651, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true

It looks like sciond.SCIONDaemon._request_paths is taking maybe 15s to respond. Don't know why yet.

@pszalach

Just found following bug executing end2end test with current code from the master.
PS in (1,13), while processing path request for (2, 26), crashes with:

Traceback (most recent call last):
  File "path_server.py", line 876, in 
    main()
  File "path_server.py", line 873, in main
    path_server.run()
  File "/home/pawel/Desktop/git/scion/infrastructure/scion_elem.py", line 183, in run
    self.handle_request(packet, addr, sock == self._local_socket)
  File "path_server.py", line 579, in handle_request
    self.dispatch_path_segment_record(pkt)
  File "path_server.py", line 139, in dispatch_path_segment_record
    self._handle_core_segment_record(pkt)
  File "path_server.py", line 347, in _handle_core_segment_record
    next_hop = self.ifid2addr[pcb.get_first_pcbm().hof.egress_if]
KeyError: 287

287 is interface number for (2, 21)->(1,11) and (1,11)->(2,21) - btw. maybe it make sense to have unique interface numbers ? So, at first glance it looks like core path is not reversed, or is constructed incorrectly (e.g. one hop is not added).

To replicate the bug few executions of scion.sh stop ; scion.sh start + PYTHONPATH=.. python3 end2end_test.py are enough for me.

I noticed that bug first time today, so @LorenzoBaesso could you start investigating that, as probably it is introduced by your latest pull request?

ps. forgot to add that sometimes end2end test works fine, so it seems that it depends on path server's cache.

The codebase really needs more unit testing. The best way to divide the work is that people pick a module, and write tests for it. I'll list all the modules that need testing, and people can sign up in the comments. I propose we start with the lib modules (ignore lib/crypto, as hopefully that goes away soon):

Unclaimed:

In progress:

Done:

• infrastructure/dns_server.py (@kormat) #126
• lib/config.py (@rev112) #133
• lib/dnsclient.py (@kormat) #250
• lib/log.py (@pratyakshs) #221
• lib/path_db.py (@prateshg) #196
• lib/path_store.py (@prateshg) #206, #214
• lib/thread.py (@mdhd) #233
• lib/topology.py (@syclops) #143, (@pratyaksh) #248
• lib/util.py (@prateshg) #201
• lib/zookeeper.py (@kormat) #101, #104
• lib/packet/ext_hdr.py (@prateshg) #124
• lib/packet/opaque_field.py (@prateshg) #132, #161
• lib/packet/packet_base.py (@pratyakshs) #134, #156, #199
• lib/packet/path.py (@pratyakshs) #138, #200, #229
• lib/packet/path_mgmt.py (@prateshg) #169
• lib/packet/pcb.py (@pratyakshs) #190
• lib/packet/scion.py (@pratyakshs) #204
• lib/packet/scion_addr.py (@mskd96) #135

Right now we're using this stack to send data in SCION:

   app protocol
-----------------
   scion
-----------------
   udp
-----------------
   ip

This is fine for the short-to-medium term, but for real-world rollout, we should be moving to:

   app protocol
-----------------
   scion
-----------------
   ip

When requesting paths, sciond is receiving multiple PCBs that are topologically equivalent but have different segment IDs. The daemon sees these as different paths even though they use the same hop sequences, so the host thinks it is using e.g. 4 paths when in fact they are all the same path.

Example PCBs:

---- pcb ----
[PathSegment]
Segment ID: b'X\xfe\xf3$\x8a3i\xa8\xe6*\t\xf2\xf4)\x87.\xc5\x94\xe3\xab\xa5\x0bC\x9c\xafoq\xd4\xee\x87\xa9\xc9'
[Info OF info: 40, up: False, TS: 1430297228, ISD ID: 1, hops: 3]
[TRC OF info: ff, TRCv: 0, IF ID: 290]

[Autonomous Domain]
[PCB Marking ad_id: 13]
ig_rev_token: b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
eg_rev_token:b'\x1dpJ\x8f\xb6K.)\xe8\x07a(;"\xf3\x08P\x91y\xe7\xc7$\x81\xc1+JQf\xfc\xf0A^'
[Support Signature OF cert_chain_version: 0, sig_len: 0, block_size: 96]
[Hop OF info: 0, exp_time: 63, ingress if: 0, egress if: 284, mac: 0]
[Info OF TD ID: 1, bwalloc_f: 0, bwalloc_r: 0]
[Signature: ]
[Autonomous Domain]
[PCB Marking ad_id: 16]
ig_rev_token: b'\nj\x13\x13(\xac:\x08\xc6\xe7\xcf`\x0f\xb7\x92\xcd9\xa3\xd0\t\x07\xfc\x1d\xe9#m\xac\xfeN\xad\xd2x'
eg_rev_token:b'\x93\xb7<=$\xc1=\xd7RC;6f\x1f\xc7P#\xcd\x89\xfe\xa1s\n\xde2E\xc6P,A\xfb8'
[Support Signature OF cert_chain_version: 0, sig_len: 0, block_size: 184]
[Hop OF info: 0, exp_time: 63, ingress if: 284, egress if: 290, mac: 0]
[Info OF TD ID: 1, bwalloc_f: 0, bwalloc_r: 0]
[Peer Marking ad_id: 15]
ig_rev_token: b'\x1b\xf73w\xa1\xa6xC\x0e\xe3\xc3\xe4\xd7\xf8;v\x1f\xb2\xaa\xa7\xb0t\xe9\xb0\xf1\x8e\r\x10\xb1?l\xf5'\eg_rev_token:b'\x93\xb7<=$\xc1=\xd7RC;6f\x1f\xc7P#\xcd\x89\xfe\xa1s\n\xde2E\xc6P,A\xfb8'
[Hop OF info: 0, exp_time: 63, ingress if: 286, egress if: 290, mac: 0]
[Support Peer OF TD ID: 1, bwalloc_f: 0, bwalloc_r: 0, bw_class: 0]
[Signature: ]
[Autonomous Domain]
[PCB Marking ad_id: 19]
ig_rev_token: b'\t\xb5\xf0\xab\xa2w\n;K\xbeq=u{E\x9a\xbb\xb6%c\xbc\x9fV\x8b]\xe6\xcdB7\x15\xad\xeb'
eg_rev_token:b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
[Support Signature OF cert_chain_version: 0, sig_len: 0, block_size: 96]
[Hop OF info: 0, exp_time: 63, ingress if: 290, egress if: 0, mac: 0]
[Info OF TD ID: 1, bwalloc_f: 0, bwalloc_r: 0]
[Signature: ]

---- pcb ----
[PathSegment]
Segment ID: b'2C\xbfJ\x102\x07\xd7\xaf"\xbb\xc0\xa3\xb9I\x8c[P\xd8\xca\xe3\n\x92]\xf1\xd8\x93\xe7\x13\x9e!$'
[Info OF info: 40, up: False, TS: 1430297278, ISD ID: 1, hops: 3]
[TRC OF info: ff, TRCv: 0, IF ID: 290]

[Autonomous Domain]
[PCB Marking ad_id: 13]
ig_rev_token: b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
eg_rev_token:b"\x0fO\xe57l9\x14\xc2\x18\x9f\xc8\xfc\xbb\x17\tC\xdd\x02\x92\xdb\xe3\t[R\xc7\xf4\xae\x82\xbag'v"
[Support Signature OF cert_chain_version: 0, sig_len: 0, block_size: 96]
[Hop OF info: 0, exp_time: 63, ingress if: 0, egress if: 284, mac: 0]
[Info OF TD ID: 1, bwalloc_f: 0, bwalloc_r: 0]
[Signature: ]
[Autonomous Domain]
[PCB Marking ad_id: 16]
ig_rev_token: b'=:\xc18;\x80\xd1G;\xbd\xd3\xe0\x01\xcbbYi"fx!\xdc*\x05z\x9eUQ\x94J\x7f\xd2'
eg_rev_token:b'\xa5\xf6F(\xb6\xc1:r;\x9d\x1b\xb0-\xd7\xea>^\xa0#\x96\xff\xb0\xa6\xeds\xbb\xfa+\xd7\x0b\x10\x89'
[Support Signature OF cert_chain_version: 0, sig_len: 0, block_size: 184]
[Hop OF info: 0, exp_time: 63, ingress if: 284, egress if: 290, mac: 0]
[Info OF TD ID: 1, bwalloc_f: 0, bwalloc_r: 0]
[Peer Marking ad_id: 15]
ig_rev_token: b"\x16\xbb\x1c\x03\x8b\xfb\x8c\xd8zL\xba\x08\xa3\x133\xfb\x06'\x94\xd6\xb4F\xb1\x82\x89~\xa0\xb1y\x13\x08C"\eg_rev_token:b'\xa5\xf6F(\xb6\xc1:r;\x9d\x1b\xb0-\xd7\xea>^\xa0#\x96\xff\xb0\xa6\xeds\xbb\xfa+\xd7\x0b\x10\x89'
[Hop OF info: 0, exp_time: 63, ingress if: 286, egress if: 290, mac: 0]
[Support Peer OF TD ID: 1, bwalloc_f: 0, bwalloc_r: 0, bw_class: 0]
[Signature: ]
[Autonomous Domain]
[PCB Marking ad_id: 19]
ig_rev_token: b'\t\xb5\xf0\xab\xa2w\n;K\xbeq=u{E\x9a\xbb\xb6%c\xbc\x9fV\x8b]\xe6\xcdB7\x15\xad\xeb'
eg_rev_token:b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
[Support Signature OF cert_chain_version: 0, sig_len: 0, block_size: 96]
[Hop OF info: 0, exp_time: 63, ingress if: 290, egress if: 0, mac: 0]
[Info OF TD ID: 1, bwalloc_f: 0, bwalloc_r: 0]
[Signature: ]

This is reminder that #263 does not include unit tests for all new methods.
Missing tests are listed as unaddressed comments.

(#129)

Should add checks on length of raw and log erroneous lengths.

Having to define PYTHONPATH everytime we do things is clunky, and prone to error. I propose that we move to using relative imports. I haven't looked at this in depth yet, so maybe there's some gotcha i'm missing, but i think it should simplify things nicely.

Currently, some tests from opaque_field_test.py and lib_packet_base_test.py fail.
@prateshg @pratyakshs

....................................................................E....................................................................................FF.EFF......................................................
======================================================================
ERROR: test.lib_packet_base_test.TestPacketBaseLen.test_basic
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/tonyo/scion_ethz/scion/test/lib_packet_base_test.py", line 159, in test_basic
    header = ExtensionHeader(b'data')
  File "/home/tonyo/scion_ethz/scion/lib/packet/ext_hdr.py", line 41, in __init__
    self.parse(raw)
  File "/home/tonyo/scion_ethz/scion/lib/packet/ext_hdr.py", line 50, in parse
    self.next_ext, self.hdr_len = struct.unpack("!BB", raw)
struct.error: unpack requires a bytes object of length 2

======================================================================
ERROR: test.opaque_field_test.TestHopOpaqueFieldPack.test_basic
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/tonyo/scion_ethz/scion/test/opaque_field_test.py", line 169, in test_basic
    ntools.eq_(hop_op_fld.pack(),bytes.fromhex('0e 2a 0a 0b 0c 0d 0e 0f'))
  File "/home/tonyo/scion_ethz/scion/lib/packet/opaque_field.py", line 162, in pack
    data += self.mac
TypeError: can't concat bytes to int

======================================================================
FAIL: test.opaque_field_test.TestHopOpaqueFieldFromValues.test_less_arg
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/tonyo/scion_ethz/scion/test/opaque_field_test.py", line 155, in test_less_arg
    ntools.eq_(hop_op_fld.mac, 0)
AssertionError: b'\x00\x00\x00' != 0

======================================================================
FAIL: test.opaque_field_test.TestHopOpaqueFieldInit.test_basic
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/tonyo/scion_ethz/scion/test/opaque_field_test.py", line 105, in test_basic
    ntools.eq_(hop_op_fld.mac, 0)
AssertionError: b'\x00\x00\x00' != 0

======================================================================
FAIL: test.opaque_field_test.TestHopOpaqueFieldParse.test_basic
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/tonyo/scion_ethz/scion/test/opaque_field_test.py", line 125, in test_basic
    ntools.eq_(hop_op_fld.mac, 0x0d0e0f)
AssertionError: b'\r\x0e\x0f' != 855567

======================================================================
FAIL: test.opaque_field_test.TestHopOpaqueFieldParse.test_len
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/tonyo/scion_ethz/scion/test/opaque_field_test.py", line 136, in test_len
    ntools.eq_(hop_op_fld.mac, 0)
nose.proxy.AssertionError: b'\x00\x00\x00' != 0
-------------------- >> begin captured logging << --------------------
root: WARNING: HOF: Data too short for parsing, len: 7
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 213 tests in 54.900s

FAILED (errors=2, failures=4)

@mskd96
#129

The following code needs to be updated:

• SCIONDaemon._api_handle_path_request in endhost.sciond
• CorePathServer.__hash__ in infrastructure.path_server
• CertChainRequest.parse in lib.packet.scion
• CertChainRequest.from_values in lib.packet.scion
• CertChainReply.parse in lib.packet.scion
• CertChainReply.from_values in lib.packet.scion
• TRCRequest.parse in lib.packet.scion
• TRCRequest.from_values in lib.packet.scion
• PathSegmentInfo.parse in lib.packet.path_mgmt
• PathSegmentInfo.pack in lib.packet.path_mgmt
• LeaseInfo.parse in lib.packet.path_mgmt
• LeaseInfo.pack in lib.packet.path_mgmt

@pszalach

https://gist.github.com/pszalach/b9e1582d9dd37bb822f5
To replicate:

diff --git a/topology/ADConfigurations.json b/topology/ADConfigurations.json
index 16ca031..217c18b 100644
--- a/topology/ADConfigurations.json
+++ b/topology/ADConfigurations.json
@@ -98,6 +98,7 @@
     "1-18": {
         "level": "LEAF",
         "cert_issuer": "1-15",
+        "path_servers": 3,
         "links": {
             "1-15": "PARENT"
         }

and run.

When end2end_test.py is run between 1,14(Intermediate node) and 1,18(Leaf node), it fails. It uses a PEER path for this case. The path is 1,14 ---> 1,15 ---> 1,18. The first one is a PEER link while the second one is a normal Parent-Child link.
I am attaching the output here:
https://gist.github.com/mskd96/0a7d44b092c2998a7a5c

AFAIK, the path information(IFID's) looks to be correct but the first hop 127.1.14.83 corresponds to the edge router connected to 1,11 which is the parent of 1,14. But, shouldn't it be the one corresponding to 1,15?
What might be the issue?

2015-05-19 14:07:57,899 INFO cs1-17-1: bound 127.1.17.21:30040
2015-05-19 14:07:57,901 INFO Connecting to Zookeeper
2015-05-19 14:07:57,925 [DEBUG](ZK state handler) Kazoo old state: startup, new state: CONNECTED
2015-05-19 14:07:57,925 [DEBUG](ZK state handler) Connection to Zookeeper succeeded
2015-05-19 14:07:57,926 INFO Started: 2015-05-19 14:07:57.926174
2015-05-19 14:07:58,226 [DEBUG](CS shared certs) Joined party, members are: ['127.1.17.21']
2015-05-19 14:08:03,598 INFO Certificate chain request received.
2015-05-19 14:08:03,598 DEBUG Certificate chain not found.
2015-05-19 14:08:03,600 INFO New certificate chain request sent.
2015-05-19 14:08:03,611 INFO Certificate chain reply received
2015-05-19 14:08:03,612 CRITICAL Exception in main process:
2015-05-19 14:08:03,613 CRITICAL Traceback (most recent call last):
2015-05-19 14:08:03,613 CRITICAL File "cert_server.py", line 420, in
2015-05-19 14:08:03,614 CRITICAL main()
2015-05-19 14:08:03,614 CRITICAL File "cert_server.py", line 416, in main
2015-05-19 14:08:03,614 CRITICAL cert_server.run()
2015-05-19 14:08:03,614 CRITICAL File "cert_server.py", line 399, in run
2015-05-19 14:08:03,614 CRITICAL SCIONElement.run(self)
2015-05-19 14:08:03,614 CRITICAL File "/home/scion/scion.git/infrastructure/scion_elem.py", line 204, in run
2015-05-19 14:08:03,614 CRITICAL self.handle_request(packet, addr, sock == self._local_socket)
2015-05-19 14:08:03,614 CRITICAL File "cert_server.py", line 384, in handle_request
2015-05-19 14:08:03,614 CRITICAL self.process_cert_chain_reply(CertChainReply(packet))
2015-05-19 14:08:03,614 CRITICAL File "cert_server.py", line 132, in process_cert_chain_reply
2015-05-19 14:08:03,614 CRITICAL "-V:" + str(tmp.certs[1].version),
2015-05-19 14:08:03,614 CRITICAL IndexError: list index out of range
2015-05-19 14:08:03,615 CRITICAL
2015-05-19 14:08:03,615 CRITICAL Exiting

@LorenzoBaesso

The page for #143 says that it's been merged by me, and that it changed a Lot of files. Except neither of those things are true. I've contacted github support, we'll see what they say.

In the mean time, PLEASE DON'T MERGE ANYTHING. Just in case.

Add support for the verification of the core segments (similar to the one for the not-core ADs).

A bug in TestPathPolicyCheckPropertyRanges._check_bandwidth prevents full coverage of _check_property_ranges.

Also, TestPathPolicyFromFile should make sure the relevant exceptions are caught and handled appropriate.

Tests that rely on a infrastructure running (e.g., end2end_test or bandwith_test) should be able to take the sender/receiver location as input, otherwise those tests might break if someone uses a different topology than the sample one.

Hash chains created for interfaces and segments have predefined length. The problem occurs when a chain gets exhausted, and some kind of versioning should be proposed to mitigate that.

My proposition is to just rotate secret keys used for tokens generation (probably anyway we will do that).
Moreover, as single core (on commodity machine) executes >3M sha256()s per second, this rotation may be infrequent.

I expected that get_hops_hash would return the same hash for all PCBs for a given path in the test system. This doesn't appear to be the case however. For example, after ~10mins of uptime the ad 1-13 beacon servers have generated nearly 900 unique path segment revocation tokens:

 893 Generating new revocation token
3550 Using existing revocation token

@shitz

(#129)

As reported by @kormat:
https://gist.github.com/kormat/6637b463a6892ab18954
I guess:

1. BS didn't propagate any path with iface
2. and tried to revoke this interface

The current beacon server revocation code doesn't seem to be used anywhere - LocalBeaconServer._process_revocation is never called (afaict).

This will also need careful consideration with regards to Zookeeper:

• communicating revocations to the other BSes in the AD
• removing now-invalidated shared PCBs
• potential for beacons for revoked paths to still be propagated when the BS master dies

Currently all scion service traffic (e.g. BSes talking to PSes) happens directly in the scion layer. This is fine in the short-term, but in the longer-term the move to ip-overlay (#300) will mean that we can no longer rely on the layer 2 (IP+UDP) providing the service address (via the UDP port number currently).

The proper fix is to move scion service traffic up 1 layer, from the scion layer to the application layer.

My suggestion is to add UDP/SCION, and have service traffic use that.

In the current master, docker.sh build terminates with:

...
Step 2 : RUN ./scion.sh topology
 ---> Running in 3cb3b5e7c3bb
Create topology, configuration, and execution files.
Traceback (most recent call last):
  File "generator.py", line 29, in 
    from dnslib.label import DNSLabel
ImportError: No module named 'dnslib'
2015/06/16 15:27:47 The command [/bin/sh -c ./scion.sh topology] returned a non-zero code: 1

I'm working on a testbed management system for SCION for my Master's thesis (including secure updates and connecting a new AD), and at the moment I'm looking for a convenient way to manage servers/processes for the current deployment model (one PC -- one or more ADs). It would be nice to have a module to keep track of all running server instances, be able to start/stop/restart them all at once, as well as having an API to do it from some other module.

The current solution with 'screen' is not flexible enough, especially when stderr and stdout are being written to log files. My suggestion is to use Supervisord (http://supervisord.org/). This lightweight tool provides a flexible way to manage processes (check status, start, restart), processes can be restarted in groups (might be useful to restart all the servers for a given AD), it also has a nice XML-RPC API and a simple web interface.

My idea is to have Supervisord running for each machine where SCION is deployed, and to have another daemon, which connects to the centralized network management interface ("AD manager", written in Django, this is a work in progress). This daemon will be responsible for receiving commands from the AD manager via some secure protocol and performing updates and/or transmitting updated topology files if necessary, and it will proxy start/stop/restart commands to the Supervisord instance.

Would be great to get some feedback on this. I really don't see any reason to stick to 'screen' any longer, so I believe it's a good idea to employ something powerful and commonly used.