DevService

- changes to RealWorldRouting
-DevelopmentService for remote development with VSCode on the simulation nodes

Author: amdfxlucas

examples/scion/S08_dev_service/README.md

@@ -0,0 +1,80 @@
+# Networking Application Development in SEED
+
+
+The DevelopmentService prepares a node for [Remote Development with VSCode](https://code.visualstudio.com/docs/remote/remote-overview).
+It installs the VSCode Remote Server on the Node during the docker image build, which allows to ssh into the running container later i.e. to debug software.
+
+As of now the primary anticipated use of the seed-emulator has been the deployment of fully developed applications into an emulated szenario i.e. to observe its behaviour.
+
+The DevService adapts the emulators primary paradigm of deploying fully develped applications into an emulated szenario,
+ to meet the need of developing i.e. P2P software to maturity in a distributed environment.
+
+The DevService allows for each individual node that it is installed on, to specify one or more Git repositories that shall be checked out
+(along with the desired filesystem path and branch) as well as the VSCode Extensions required for the projects software stack
+(i.e. golang.Go for the SCION-proto implementation which is written in Go)
+The DevService takes care to install the entirety of build and analysis tools that are needed for a given programming language at docker image build time (for Go this being i.e. the compiler, language server gopls, debugger delve) so that no annoying time delay arises once the emulator is running and you want to attach to a container.
+Any specified Git repositories are checked out on its own separate docker volume, for the changes to persist between runs of the simulator in case one forgets to push.
+
+Software development requires a 'real Internet' connection of the container be it to git push/pull or fetch project dependencies for a build (i.e. go get, cargo build etc. )
+ This is achieved by promoting the nodes default gateway ( router node ) in the simulation into a 'RealWorldRouter' (which has access to the `000_svc` service network)
+Use of a separate service network inhibits 'short-circuiting' the simulated network topology (i.e. any crosstalk past the intended network topo among nodes)
+
+
+
+### Usage
+This example is meant to showcase how SEED can be made into a testbed for development of features for SCION.
+This is mainly achieved by two steps:
+- deploy your custom SCION stack inside the simulation
+  This is achieved by setting the `SetupSpecification` option on the `ScionRouting`layer and
+  make it point to the specific repo checkout you want to use.
+  As a consequence a named docker volume containing the binary distributables is mounted into every node in the simulation
+  under the `/bin/scion` path.
+
+- install a GolangDevelopmentServer on every node in the emulation
+  This turns every simulation node into a Golang development environment (with go, git, delve, gopls, vscode-server),
+  by mounting a bunch of docker volumes into each container containing the installation.
+  (they'll even share the module/build cache)
+   For each repo that you check out a named docker volume will be generated that accommodates the code and is shared between all nodes.
+   If you now set up your project build-task to output the distributables into `/bin/scion`, you just need to restart the simulation docker containers,
+   to run and test a SCION stack with your latest changes in place.
+   Moreover changes done to the code in one container, are visible to all others (if checkout mode is `AccessMode.shared` ),
+    so it doesn't matter on which of the nodes you trigger a recompile. The changes will take effect on all simulation nodes.
+
+
+```
+    spec = SetupSpecification.LOCAL_BUILD(
+            CheckoutSpecification(
+                mode = "build",
+                git_repo_url = "https://github.com/johndoe/scion.git", # your SCION fork
+                checkout = "v0.12.0" # could be tag, branch or commit-hash
+            ))
+    routing = ScionRouting(setup_spec=OptionRegistry().scion_setup_spec(spec))
+```
+
+
+```
+    devsvc = GolangDevService('jane.doe', '[email protected]')
+    repo_url = 'https://github.com/scionproto/scion.git'
+    repo_branch = 'v0.12.0'
+    repo_path = '/home/root/repos/scion'
+
+    svc = devsvc.install(f'dev_152_cs1')
+    svc.checkoutRepo(repo_url, repo_path, repo_branch, AccessMode.shared)
+    emu.addBinding(Binding(f'dev_152_cs1', filter=Filter(nodeName=as152_cs1.getName(), asn=152)))
+
+    svc3 = devsvc.install(f'dev_153_cs1')
+    svc3.checkoutRepo(repo_url, repo_path, repo_branch, AccessMode.shared)
+    emu.addBinding(Binding(f'dev_153_cs1', filter=Filter(nodeName=as153_cs1.getName(), asn=153)))
+
+```
+
+
+To do any development you first have to bring up the simulation `docker compose up` and then attach to one of the containers with DevServer installation
+(i.e. from within VSCode IDE via the [RemoteDevelopment Extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack) ).
+You can attach to multiple nodes at the same time and i.e. send a packet from the integrated terminal in one window,
+ and hit a breakpoint set in the border router of any on-path AS once the packet eventually arrives there and step through to see if its correctly processed the way you intended.
+
+ ![Attach to Emulator](media/attach_to_emu_container.png)
+
+
+ ![Develop your own SCION features in SEED](media/inside_dev_container.png)

examples/scion/S08_dev_service/media/attach_to_emu_container.png

@@ -4,10 +4,10 @@
 
 from seedemu.services import GolangDevService, AccessMode
 from seedemu.core import Emulator, Binding, Filter
-  
+
 from seedemu.compiler import Docker
-from seedemu.core import Emulator
-from seedemu.layers import ScionBase, ScionRouting, ScionIsd, Scion
+from seedemu.core import Emulator, OptionRegistry
+from seedemu.layers import ScionBase, ScionRouting, ScionIsd, Scion, CheckoutSpecification, SetupSpecification
 from seedemu.layers.Scion import LinkType as ScLinkType
 
 from seedemu.compiler import Docker, Platform
@@ -32,19 +32,25 @@ def run(dumpfile = None):
         else:
             print(f"Usage:  {script_name} amd|arm")
             sys.exit(1)
-    
+
     # Initialize
     emu = Emulator()
     base = ScionBase()
-    routing = ScionRouting()
+    spec = SetupSpecification.LOCAL_BUILD(
+            CheckoutSpecification(
+                mode = "build",
+                git_repo_url = "https://github.com/scionproto/scion.git",
+                checkout = "v0.12.0" # could be tag, branch or commit-hash
+            ))
+    routing = ScionRouting(setup_spec=OptionRegistry().scion_setup_spec(spec))
     scion_isd = ScionIsd()
     scion = Scion()
 
     devsvc = GolangDevService('jane.doe', '[email protected]')
     repo_url = 'https://github.com/scionproto/scion.git'
     repo_branch = 'v0.12.0'
     repo_path = '/home/root/repos/scion'
-                 
+
 
 
     # SCION ISDs
@@ -58,7 +64,7 @@ def run(dumpfile = None):
     scion_isd.addIsdAs(1, 150, is_core=True)
     as150.createNetwork('net0')
     as150.createControlService('cs1').joinNetwork('net0')
-    as150_router = as150.createRealWorldRouter('br0', prefixes=['10.150.0.0/24'])# the 'auto' gen prefix of net0
+    as150_router = as150.createRealWorldRouter('br0', prefixes=['0.0.0.0/1', '128.0.0.0/1'])
     # expectation: hosts from within AS150 can ping outside world i.e. 8.8.8.8
     #   Hosts in the other ASes can't!!
     as150_router.joinNetwork('net0').joinNetwork('ix100')
@@ -83,8 +89,8 @@ def run(dumpfile = None):
     scion_isd.addIsdAs(1, 153, is_core=False)
     scion_isd.setCertIssuer((1, 153), issuer=150)
     as153.createNetwork('net0')
-    as153_cs1 = as153.createControlService('cs1').joinNetwork('net0') 
-    
+    as153_cs1 = as153.createControlService('cs1').joinNetwork('net0')
+
     as153_router = as153.createRouter('br0')
     as153_router.joinNetwork('net0')
     as153_router.crossConnect(150, 'br0', '10.50.0.3/29')
@@ -117,7 +123,7 @@ def run(dumpfile = None):
         emu.render()
 
         ###############################################################################
-        # Compilation 
+        # Compilation
 
         emu.compile(Docker(platform=platform), './output', override=True)
 

examples/scion/S08_dev_service/media/inside_dev_container.png

@@ -305,6 +305,9 @@ def getRemoteAccessProvider(self) -> RemoteAccessProvider:
     def getExternalConnectivityProvider(self) -> ExternalConnectivityProvider:
         return self.__ecp
 
+    def getExternalConnectivityProvider(self) -> ExternalConnectivityProvider:
+        return self.__ecp
+
     def print(self, indent: int) -> str:
         out = ' ' * indent
         out += 'Network {} ({}):\n'.format(self.__name, self.__type)

examples/scion/S08_real_world/scion_dev_service.py renamed to examples/scion/S08_dev_service/scion_dev_service.py

@@ -1184,7 +1184,7 @@ class RealWorldRouterMixin():
     @todo real world access.
     """
 
-    __realworld_routes: List[str]
+    __realworld_routes: List[Tuple[str, Optional[List[str]]]]
     __sealed: bool
     __hide_hops: bool
 
@@ -1198,18 +1198,18 @@ def initRealWorld(self, hideHops: bool):
         self.__hide_hops = hideHops
         self.addSoftware('iptables')
 
-    def addRealWorldRoute(self, prefix: str) -> 'RealWorldRouter':
+    def addRealWorldRoute(self, prefix: str, route_clientele: str = None) -> 'RealWorldRouter':
         """!
         @brief Add real world route.
 
-        @param prefix prefix.
-
+        @param prefix prefix i.e. ['0.0.0.0/1', '128.0.0.0/1'] for the 'real' real whole world global internet
+        @param route_clientele the network/s who shall be able to route to the given 'real world' prefix
         @throws AssertionError if sealed.
 
         @returns self, for chaining API calls.
         """
         assert not self.__sealed, 'Node sealed.'
-        self.__realworld_routes.append(prefix)
+        self.__realworld_routes.append( (prefix, route_clientele) )
 
         return self
 
@@ -1219,10 +1219,10 @@ def getRealWorldRoutes(self) -> List[str]:
 
         @returns list of prefixes.
         """
-        return self.__realworld_routes
+        return [r for (r, _) in self.__realworld_routes]
 
     # called in RoutingLayer::render() - That is AFTER ::configure() :)
-    # where its been decided if this node has to run BIRD routing daemon 
+    # where its been decided if this node has to run BIRD routing daemon
     # because it is connected to more than one local(direct) network
     def seal(self, svc_net: Network):
         """!
@@ -1238,10 +1238,17 @@ def seal(self, svc_net: Network):
         # position 0-1 is '/interface_setup' (and chmod +x)
         self.insertStartCommand(2, '/rw_configure_script')
         self.insertStartCommand(2, 'chmod +x /rw_configure_script')
-        
-        for prefix in self.__realworld_routes:
+
+        for prefix, route_clientele in self.__realworld_routes:
+            if route_clientele != None:
+                if isinstance(route_clientele, list):
+                    for src in route_clientele:
+                        self.appendFile('/rw_configure_script', 'iptables -t nat -A POSTROUTING -d {} -s {} -j MASQUERADE\n'.format(prefix, src))
+                else:
+                    self.appendFile('/rw_configure_script', 'iptables -t nat -A POSTROUTING -d {} -s {} -j MASQUERADE\n'.format(prefix, route_clientele))
             # nat matched only
-            self.appendFile('/rw_configure_script', 'iptables -t nat -A POSTROUTING -d {} -j MASQUERADE\n'.format(prefix))
+            else:
+                self.appendFile('/rw_configure_script', 'iptables -t nat -A POSTROUTING -d {} -j MASQUERADE\n'.format(prefix))
 
             if self.__hide_hops:
                 # remove realworld hops
@@ -1251,15 +1258,14 @@ def seal(self, svc_net: Network):
         # some might run SCION BR instead
         if 'bird2' in self.getSoftware():
             # if this check is too dirty/hacky, we could use Attributes like in: "if hasattr(self, '__sealed'):" but this is still hacky
-            self.appendFile('/rw_configure_script', )
             fill_placeholder = """\
             gw="`ip rou show default | cut -d' ' -f3`"
             sed -i 's/!__default_gw__!/'"$gw"'/g' /etc/bird/bird.conf
             """
             self.appendFile('/rw_configure_script', fill_placeholder)
 
             self.addTable('t_rw')
-            statics = '\n    ipv4 { table t_rw; import all; };\n    route ' + ' via !__default_gw__!;\n    route '.join(self.__realworld_routes)
+            statics = '\n    ipv4 { table t_rw; import all; };\n    route ' + ' via !__default_gw__!;\n    route '.join(self.getRealWorldRoutes())
             statics += ' via !__default_gw__!;\n'
             self.addProtocol('static', 'real_world', statics)
             self.addTablePipe('t_rw', 't_bgp', exportFilter = 'filter { bgp_large_community.add(LOCAL_COMM); bgp_local_pref = 40; accept; }')
@@ -1287,7 +1293,9 @@ def print(self, indent: int) -> str:
         return out
 
 def promote_to_real_world_router(node: Node, hideHops: bool):
-    """Dynamically inject RealWorldRouterMixin into a Node instance"""
+    """!@brief Dynamically inject RealWorldRouterMixin into a Node instance
+                to augment it by RealWorld routing capabilities
+    """
 
     if not isinstance(node, RealWorldRouterMixin):  # Prevent double-mixing
         node.__class__ = type("RealWorldRouter", (RealWorldRouterMixin, node.__class__), {})
@@ -1304,7 +1312,7 @@ class ScionRouterMixin():
 
     __interfaces: Dict[int, Dict]  # IFID to interface
     __next_port: int               # Next free UDP port
-    
+
     # Never been used anyway
     #def __init__(self):
     #   super().__init__()
@@ -1357,7 +1365,7 @@ def print(self, indent: int) -> str:
         out += ' ' * indent
         out += 'SCION border router'
         return out
-    
+
 def promote_to_scion_router(node: Node):
     """Dynamically inject ScionRouterMixin into a Node instance"""
     if not isinstance(node, ScionRouterMixin):  # Prevent double-mixing

examples/scion/S08_real_world/README.md

@@ -170,7 +170,6 @@ def createRealWorldRouter(self, name: str, hideHops: bool = True, prefixes: List
         # this is fine, since the ScionRouter will be 'mixed-in' in later in the ::configure() step,
         # if the router is actually found to be a border router
         return super().createRealWorldRouter(name, hideHops, prefixes)
-
     def registerNodes(self, emulator: Emulator):
         """!
         @copydoc AutonomousSystem.registerNodes()

seedemu/core/Network.py

@@ -163,7 +163,7 @@ def render(self, emulator: Emulator):
         hit: bool = False
         for ((scope, type, name), obj) in reg.getAll().items():
             # make sure that on each externaly connected net (those with at least one host who requested it)
-            #  (I):  there is at least one RealWorldRouter 
+            #  (I):  there is at least one RealWorldRouter
             #  (II): the RWR is the default gateway of the requesters on this net
             if type == 'net' and obj.getType() == NetworkType.Local:
                 if (p := obj.getExternalConnectivityProvider() ):
@@ -172,7 +172,9 @@ def render(self, emulator: Emulator):
                    for r in rwr_candidates:
                        r = promote_to_real_world_router(r, False)
                        route = obj.getPrefix()
-                       r.addRealWorldRoute(route)
+                       # only for hosts on THIS network ('route') the RWA is provided
+                       r.addRealWorldRoute('0.0.0.0/1', str(route))
+                       r.addRealWorldRoute('128.0.0.0/1', str(route))
                    for h, gw in new_gateway_constraints.items():
                        assert h not in gateway_constraints, 'multihomed host ?!'
                        gateway_constraints[h] = gw
@@ -187,7 +189,7 @@ def render(self, emulator: Emulator):
             if type == 'rnode':
                 rnode: Router = obj
                 if issubclass(rnode.__class__, RealWorldRouterMixin): # could also be ScionRouter which needs RealWorldAccess
-                    
+
                     # this is an exception - Only for service net (not part of simulation)
                     rnode._Node__joinNetwork(svc_net)
                     [l, b, d] = svc_net.getDefaultLinkProperties()
@@ -210,7 +212,7 @@ def render(self, emulator: Emulator):
                 else:
                     cur_scope = ScopedRegistry(scope, reg)
                     candidates = cur_scope.getByType('rnode')
-                
+
 
                 for router in candidates:
                     if rif != None: break