(Mostly) Documentation improvements

* document options to run network with couchdb and/or hl explorer
* document how to run docker-compose behind proxy and improve docu related to proxies
* some minimal extensions of the demo docu and a reference thereof in the top README
* a few improvements in network setup script

Signed-off-by: michael steiner <[email protected]>

Author: g2flyer

README.md

@@ -110,34 +110,41 @@ your [development environment](https://hyperledger-fabric.readthedocs.io/en/rele
 
 Moreover, we assume that you are familiar with the Intel SGX SDK.
 
-### Intel SGX SDK and SSL
+### Intel SGX
 
-Fabric Private Chaincode requires the Intel [SGX SDK](https://github.com/intel/linux-sgx) and
-[SGX SSL](https://github.com/intel/intel-sgx-ssl) to build the main components of our framework and to develop and build
-your first private chaincode.     
-
-Install the Intel SGX software stack for Linux (including the SGX driver, the SGX SDK, and the SGX Platform Software
-(PSW)) by following the official [documentation](https://github.com/intel/linux-sgx). Please make sure that you use the
-SDK version as denoted above in the list of requirements. 
-
-Moreover, if you don't have SGX hardware available you can also install the SGX SDK only and use simulation mode by
-setting `SGX_MODE=SIM` in your environment. In this case, also make sure that simulation mode is set when building
-and installing [SGX SSL](https://github.com/intel/intel-sgx-ssl#available-make-flags). Note that the simulation mode is
-for developing purpose only and does not provide any security guarantees. 
+To run Fabric Private Chaincode in secure mode, you need an SGX-enabled
+hardware as well corresponding OS support.  However, even if you don't
+have SGX hardware available, you still can run FPC in simulation mode by
+setting `SGX_MODE=SIM` in your environment.
 
-Once you have installed the SGX SDK and SSL for SGX SDK please double check that ``SGX_SDK`` and ``SGX_SSL`` variables
-are set correctly in your environment. 
+Note that the simulation mode is for developing purpose only and does
+not provide any security guarantees.
 
 Notice: by default the project builds in hardware-mode SGX, ``SGX_MODE=HW`` as defined in `<absolute-project-path>/fabric-private-chaincode/config.mk` and you can
 explicitly opt for building in simulation-mode SGX, ``SGX_MODE=SIM``. In order to set non-default values for install
 location, or for building in simulation-mode SGX, you can create the file `<absolute-project-path>/fabric-private-chaincode/config.override.mk` and override the default
 values by defining the corresponding environment variable.
 
-#### Register with Intel Attestation Service (IAS) if using SGX_MODE=HW
+If you run SGX in __simulation mode only__, you can skip below
+sections and jump right away to [Setup your Development
+Environment](#setup-your-development-environment).
+
+Note that you can always come back here when you want a setup with SGX
+hardware-mode later after having tested with simulation mode.
+
+
+#### Install SGX Operation System Support
+
+Install the Intel SGX software stack for Linux (including the SGX
+driver and the SGX Platform Software (PSW)) by following the official
+[documentation](https://github.com/intel/linux-sgx).
+
+
+#### Register with Intel Attestation Service (IAS)
+
+We currently support EPID-based attestation and  use the Intel's
+Attestation Service to perform attestation with chaincode enclaves.
 
-We use Intel's Attestation Service to perform attestation with chaincode enclaves. If you run SGX in __simulation mode__
-only, you can skip this section and come back when you want setup with SGX hardware-mode.
- 
 What you need:
 
 * a Service Provider ID (SPID)
@@ -213,7 +220,7 @@ Optional: to do a clean build do the following within the container
 <docker-root>:project/src/github.com/hyperledger-labs/fabric-private-chaincode# make build
 ```
 Now you are ready to start development.  Go to the [Develop Your First Private Chaincode
-](#developing-your-first-private-chaincode) section.
+](#your-first-private-chaincode) section.
 
 ### Option 2: Setting up your system to do local development
 
@@ -243,6 +250,26 @@ Make sure that you have the following required dependencies installed:
 
 * [PlantUML](http://plantuml.com/) including Graphviz (for building documentation only) 
 
+#### Intel SGX SDK and SSL
+
+Fabric Private Chaincode requires the Intel [SGX SDK](https://github.com/intel/linux-sgx) and
+[SGX SSL](https://github.com/intel/intel-sgx-ssl) to build the main components of our framework and to develop and build
+your first private chaincode.     
+
+Install the Intel SGX software stack for Linux by following the
+official [documentation](https://github.com/intel/linux-sgx). Please make sure that you use the
+SDK version as denoted above in the list of requirements. 
+
+For SGX SSL, just follow the instructions on the [corresponding
+github page](https://github.com/intel/intel-sgx-Sal). In case you are
+building for simulation mode only and do not have HW support, you
+might also want to make sure that [simulation mode is set](https://github.com/intel/intel-sgx-ssl#available-make-flags)
+when building and installing it.
+
+Once you have installed the SGX SDK and SSL for SGX SDK please double check that ``SGX_SDK`` and ``SGX_SSL`` variables
+are set correctly in your environment. 
+
+
 #### Protocol Buffers
 
 We use *nanopb*, a lightweight implementation of Protocol Buffers, inside the ledger enclave to parse blocks of 
@@ -330,14 +357,23 @@ Building the project requires docker. We do not recommend to run `sudo make`
 to resolve issues with mis-configured docker environments as this also changes your `$GOPATH`. Please see hints on
 [docker](#docker) installation above.
 
+The makefiles do not ensure that docker files are always rebuild to
+match the latest version of the code in the repo.  If you suspect you
+have an issue with outdated docker images, you can run `make clobber
+build` which forces a rebuild.  It also ensures that all other
+downlad, build or test artifacts are scrubbed from your repo and might
+help overcoming other problems. Be advised that that the rebuild can
+take a fair amount of time.
+
 ##### Working from behind a proxy
 
 The current code should work behind a proxy assuming
   * you have defined the corresponding environment variables (i.e.,
-  `http_proxy`, `https_proxy` and, potentially, `no_proxy`) properly
-  defined
+    `http_proxy`, `https_proxy` and, potentially, `no_proxy`) properly, and
   * docker (daemon & client) is properly set up for proxies as
-    outlined in the Docker documentation for [clients](https://docs.docker.com/network/proxy/) and the [daemon](https://docs.docker.com/config/daemon/systemd/#httphttps-proxy).
+    outlined in the Docker documentation for
+    [clients](https://docs.docker.com/network/proxy/) and the
+    [daemon](https://docs.docker.com/config/daemon/systemd/#httphttps-proxy).
 If you run Ubuntu 18.04, make sure you run docker 18.09 or later. Otherwise you will run into problems with DNS resolution inside the container.
 
 You will also require a recent version of docker-compose. In particular, the docker-compose from ubuntu 18.04
@@ -412,13 +448,15 @@ will create dummy files. In case you switch later to HW mode without
 configuring these files correctly for HW mode, this will result in
 above error.
 
-## Developing your first private chaincode
+## Developing with Fabric Private Chaincode
 
+### Your first private chaincode
 Create, build and test your first private chaincode with this [tutorial](examples/README.md).
 
 ### A Complete Application
 For an end-to-end application demonstrating the potential of FPC, check out our [Clock Auction Demo Application](demo/README.md).
 
+
 ## Documentation
 
 To build documentation, you will have to install `java` and download `plantuml.jar`. Either put `plantuml.jar` into

demo/README.md

@@ -45,8 +45,8 @@ FPC peer cli hides this name mapping, you will have to manually prefix the
 chaincode name in `config.json`, hence the default `ecc_auctioncc`.
 
 Both the frontend and fabric-gateway [expose ports](docker-compose.yml) and are
-accessible on the host machine. The frontend can be accessed at `localhost:5000`
-and the fabric-gateway can be accessed at `localhost:3000`.
+accessible on the host machine. The frontend can be accessed at [localhost:5000](http://localhost:5000)
+and the fabric-gateway can be accessed at [localhost:3000](http://localhost:3000).
 
 Below is the script's help text.
 
@@ -59,7 +59,9 @@ startFPCAuctionNetwork.sh [options]
    compliant auction chaincode($FPC_PATH/demo/chaincode/fpc), register auction users, and bring up
    both the fabric-gatway & frontend UI. If the fabric-gateway and frontend UI docker images have
    not previously been built it will build them, otherwise the script will reuse the images already
-   existing. The FPC chaincode will not be built unless specified by the flag --build-cc.
+   existing. The FPC chaincode will not be built unless specified by the flag --build-cc.  
+   By calling the script with both build options, you will be able to run the demo without having
+   to build the whole FPC project (e.g., by calling `make` in $FPC_PATH).
 
    options:
        --build-cc:
@@ -90,6 +92,31 @@ scripts/teardown.sh
 in the FPC Network scripts. If you run it with the `--clean-slate` flag the script
 will delete all the unused volumes and chaincode images.
 
+
+### Scripting
+
+To facilitate  demonstrations and also to help in testing, you can specify a scenario script defining the
+actions of the different parties and execute it using the command [scenario-run.sh](client/scripting/scenario-run.sh).  
+Below is the script's help text.
+```
+scenario-run.sh [--help|-h|-?] [--bootstrap|-b] [--dry-run|-d] [--non-interactive|-n] [--skip-delay|-s] [--mock-reset|-r] <script-file>
+    Run the demo scenario codified in the passed script file.
+    - If you pass option --bootstrap, it will also first bring up an FPC network
+      and tear it down at the end; otherwise it assumes you have already
+      a running setup ...
+    - option --dry-run/-d will just display all requests but not execute/submit
+      any of them
+    - option --non-interactive/-n will case _all_ requests to be submited even
+      requests from submit_manual.  This allows you to easily validate all
+      json files, even when some steps would be manual in an actual scenario
+    - option --skip-delay/-s allows you ignore all delays to speed-up demo
+    - option --mock-reset/-r will try reset the mock-server at the beginning
+      (obviously, this won't work if you run against the fabric-gateway backend;
+       to achieve the equivalent for fabric-gateway, use option --bootstrap)
+```
+
+An example of a scenario can be found in [demo/scenario](scenario).
+
 ## Manually Bring Up The Components
 
 ### 1. FPC Network

demo/client/scripting/lib/dsl.sh

@@ -12,7 +12,10 @@
 
 CLI="${DEMO_CLIENT_SCRIPTS_DIR}/cli"
 
-
+if [ ! -x ${CLI} ]; then
+    # cli does not exist, try to build it
+    make -C ${DEMO_CLIENT_SCRIPTS_DIR} cli || die "command $LCI did not exist and could not be built"
+fi
 
 
 #  Simple "DSL" to script auction scenarios

demo/scripts/startFPCAuctionNetwork.sh

@@ -55,16 +55,18 @@ for var in "$@"; do
     shift
 done
 
-export DEMO_SCRIPTS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
-export FPC_PATH="${FPC_PATH:-${DEMO_SCRIPTS_DIR}/../..}"
-export DEMO_ROOT="${DEMO_SCRIPTS_DIR}/.."
+export DEMO_SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+export DEMO_ROOT="${DEMO_SCRIPT_DIR}/.."
+DEMO_DOCKER_COMPOSE=${DEMO_SCRIPT_DIR}/../docker-compose.yml
+
+export FPC_PATH="${FPC_PATH:-${DEMO_SCRIPT_DIR}/../..}"
 
 # SCRIPT_DIR is the docker compose script dir that needs to be defined to source environment variables from the FPC Network
 export SCRIPT_DIR=${FPC_PATH}/utils/docker-compose/scripts
 . ${SCRIPT_DIR}/lib/common.sh
 
 # Cleanup any previous iterations of the demo
-"${SCRIPT_DIR}/teardown.sh"
+"${DEMO_SCRIPT_DIR}/teardown.sh"
 
 # Generate the necessary credentials and start the FPC network
 "${SCRIPT_DIR}/generate.sh"
@@ -80,15 +82,15 @@ fi
 if $BUILD_CLIENT; then
     echo ""
     echo "Building Fabric Gateway and Frontend UI"
-    COMPOSE_IGNORE_ORPHANS=true ${DOCKER_COMPOSE_CMD} -f ${DEMO_ROOT}/docker-compose.yml build
+    COMPOSE_IGNORE_ORPHANS=true ${DOCKER_COMPOSE_CMD} -f ${DEMO_DOCKER_COMPOSE} build
 fi
 
 # Start the FPC Network using utils/docker-compose scripts
 echo "Starting the FPC Network"
 "${SCRIPT_DIR}/start.sh"
 
 # Install and Instantiate Auction Chaincode
-"${DEMO_SCRIPTS_DIR}/installCC.sh"
+"${DEMO_SCRIPT_DIR}/installCC.sh"
 
 # Register Users
 "${DEMO_ROOT}/client/backend/fabric-gateway/registerUsers.sh"
@@ -97,4 +99,4 @@ echo "Starting the FPC Network"
 # Since the new containers are going to use the same network as the FPC network, docker-compose
 # typically throws a warning as it sees containers using the network. To quiet the warning set
 # COMPOSE_IGNORE_ORPHANS to true.
-COMPOSE_IGNORE_ORPHANS=true ${DOCKER_COMPOSE_CMD} -f ${DEMO_ROOT}/docker-compose.yml up -d auction_client auction_frontend
+COMPOSE_IGNORE_ORPHANS=true ${DOCKER_COMPOSE_CMD} -f ${DEMO_DOCKER_COMPOSE} up -d auction_client auction_frontend

demo/scripts/teardown.sh

@@ -51,7 +51,7 @@ if $CLEAN_SLATE; then
 else
 	DOWN_OPT=""
 fi
-COMPOSE_IGNORE_ORPHANS=true docker-compose -f ${DEMO_DOCKER_COMPOSE} kill && COMPOSE_IGNORE_ORPHANS=true docker-compose -f ${DEMO_DOCKER_COMPOSE} down ${DOWN_OPT}
+COMPOSE_IGNORE_ORPHANS=true ${DOCKER_COMPOSE_CMD} -f ${DEMO_DOCKER_COMPOSE} kill && COMPOSE_IGNORE_ORPHANS=true docker-compose -f ${DEMO_DOCKER_COMPOSE} down ${DOWN_OPT}
 
 if $CLEAN_SLATE; then
     "${SCRIPT_DIR}/teardown.sh" --clean-slate

utils/Makefile

@@ -8,4 +8,4 @@ include $(TOP)/build.mk
 SUB_DIRS = docker fabric docker-compose
 
 all build test clean clobber:
-	$(foreach DIR, $(SUB_DIRS), $(MAKE) -C $(DIR) $@;)
+	$(foreach DIR, $(SUB_DIRS), $(MAKE) -C $(DIR) $@ || exit ;)

utils/docker-compose/README.md

@@ -32,7 +32,16 @@ is used. Otherwise it will use `core.yaml` and the regular peer image.
    scripts/start.sh
    ```
    This will create all necessary installation artifacts and start the
-   network. For more information in the steps involved, continue
+   network. 
+   If you set the environment variable `USE_EXPLORER` to `true`, the network will include
+   and start the [Hyperledger Explorer](https://www.hyperledger.org/projects/explorer) on 
+   [port 8090](http://localhost:8090). This will enable you to inspect the networks, 
+    e.g., processed transactions.
+   If you set the environment variable `USE_COUCHDB` to `true`, the peer will use couchdb
+   to store the local version of the ledger and you can inspect the peer's ledger state
+   on [port 5984](http://localhost:5984) (login as user `admin` with password `adminPassword`).
+
+   For more information in the steps involved, continue
    reading the following section. Otherwise, you can skip to the
    Section on [Chaincode Installation](#deploying-your-fpc-chaincode).
 

utils/docker-compose/network-config/docker-compose-couchdb.yml

@@ -7,8 +7,7 @@
 version: '2'
 
 services:
-  couchdb0:
-    container_name: couchdb0
+  couchdb0.org1.example.com:
     image: couchdb:2.3
     # Populate the COUCHDB_USER and COUCHDB_PASSWORD to set an admin user and password
     # for CouchDB.  This will prevent CouchDB from operating in an "Admin Party" mode.
@@ -25,12 +24,12 @@ services:
   peer0.org1.example.com:
     environment:
       - CORE_LEDGER_STATE_STATEDATABASE=CouchDB
-      - CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdb0:5984
+      - CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdb0.org1.example.com:5984
       # The CORE_LEDGER_STATE_COUCHDBCONFIG_USERNAME and CORE_LEDGER_STATE_COUCHDBCONFIG_PASSWORD
       # provide the credentials for ledger to connect to CouchDB.  The username and password must
       # match the username and password set for the associated CouchDB.
       - CORE_LEDGER_STATE_COUCHDBCONFIG_USERNAME=
       - CORE_LEDGER_STATE_COUCHDBCONFIG_PASSWORD=
     depends_on:
-      - couchdb0
+      - couchdb0.org1.example.com
 

utils/docker-compose/scripts/start.sh

@@ -14,8 +14,8 @@ export SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && p
 
 # test pre-conditions and try to remediate
 #
-# - existance of FPC peer
 if [[ ! $USE_FPC = false ]]; then
+    # - existance of FPC peer
     FPC_PEER_NAME="hyperledger/fabric-peer-fpc"
     if [ -z "$(docker images | grep ${FPC_PEER_NAME})" ]; then
 	echo "FPC peer container image '${FPC_PEER_NAME}' does not exist, try to build it ..."
@@ -24,6 +24,20 @@ if [[ ! $USE_FPC = false ]]; then
 	make peer || die "can't build peer"
 	popd
     fi
+    # - existance of boilerplate
+    BOILERPLATE_NAME="hyperledger/fabric-private-chaincode-boilerplate-ecc"
+    if [ -z "$(docker images | grep ${BOILERPLATE_NAME})" ]; then
+	echo "FPC boilerplate container image '${BOILERPLATE_NAME}' does not exist, try to build it ..."
+	pushd "${FPC_PATH}/" || die "can't go to fpc-sdk and boilerplate build location"
+	make fpc-sdk || die "can't build fpc sdk"
+	popd
+	pushd "${FPC_PATH}/utils/docker" || die "can't go to docker build location"
+	make || die "can't build docker base images"
+	popd
+	pushd "${FPC_PATH}/ecc" || die "can't go to fpc-sdk and boilerplate build location"
+	make docker-boilerplate-ecc || die "can't build boilerplate"
+	popd
+    fi
 fi
 # - generated crypto-config files
 #   test that we have generated crypto-config. Otherwise below up

utils/docker/Makefile

@@ -46,9 +46,9 @@ clobber:
 		dev-* \
 		dev_test-* \
 	        ${FPC_PEER_DOCKER_NAMESPACE} \
+		$(FPC_DOCKER_NAMESPACE)-cc-builder \
 		$(FPC_DOCKER_NAMESPACE)-dev \
 		$(FPC_DOCKER_NAMESPACE)-ccenv \
-		$(FPC_DOCKER_NAMESPACE)-cc-builder \
 		$(FPC_DOCKER_NAMESPACE)-base \
 	; do \
 		IMAGES=$$(${DOCKER} images $${imgs} -q); \