Outline

June 26, 2020 ยท View on GitHub

The following document walks you through the steps required to set up for development on all Web of Needs components, which are Java Spring applications deployed to a Tomcat server via Eclipse. Maven is used for dependency managment and building.

Building

1. Java SDK version 8 or newer

Sources:

Things to mind:

  • Make sure your $JAVA_HOME environment variable points to the folder it's installed in.
  • Make sure the Java-Binaries are on the system-path/$PATH
    • On Linux-distributions using alternatives, check alternatives --list | grep java and make sure you have the right SDK selected. You can change the configuration via sudo alternatives --config <alternative-name>

2. Eclipse for Java Enterprise Developers (4.7 Oxygen or newer)

Sources:

If you installed from your package manager, ensure the following addons are also installed:

  • Eclipse Java EE Developer Tools
  • Eclipse Web Developer Tools
  • m2e Maven Integration

You can check in Help >> Install New Software >> "What's already installed". If anything isn't you can select "Work With: All Available Sites" and then search, select and install the missing addons.

3. Install Maven (3.3 or newer)

Sources:

If it's installed you should be able to run mvn -version on the command-line.

Make sure it's on the system $PATH.

4. Git Clone to Your Eclipse Workspace

Using your terminal (we recommend Git SCM if you're on Windows) run:

git clone https://github.com/researchstudio-sat/webofneeds.git

5. Import the Maven-project in Eclipse

File >> Import >> Existing Maven Project >> select the (folder with) the pom.xml webofneeds/pom.xml

Troubleshooting: If you don't have the "Existing Maven Project"-option, make sure you have the addons mentioned in the eclipse section above.

Deactivate eclipse autobuild

Window >> Preferences >> General >> Workspace >> Build >> uncheck "Build automatically"

This increases performance.

Use the provided code style

Window >> Preferences >> Java >> Code Style >> Formatter

  1. Click Import
  2. Select the file webofneeds/won-buildtools/src/main/resources/eclipse/formatter.xml
  3. Click Apply and Close

If you want to commit to our repositories, this step we'd ask you to do this.

7. Maven install

In Eclipse:

  1. Right-click the webofneeds project in the Project Explorer Tab on the left side >> Maven >> Select Maven Profiles. Check skip-tests.
  2. right-click "webofneeds" in project explorer >> Run As >> Maven Install

Or in your terminal:

mvn install -P skip-tests

This is required e.g. so the project-local npm exists for the next step, the required WAR-archives and JAR-files exist, etc.

Further information on what you can do with the maven-configuration can be found in ./maven.md.

8. On Windows: build-tools for project-local npm

When you the maven-install reaches the owner-application, the maven-frontend plugin will install it's own version of npm. Using that run the following with admin permissions:

<PROJECT_ROOT>/webofneeds/won-owner-webapp/src/main/webapp/node/npm install -g windows-build-tools

If your Windows User does not have admin permissions, installing windows-build-tools will unfortunately install them only for that user. (Further info at https://github.com/researchstudio-sat/webofneeds/pull/1743) The Visual Studio build tools are installed correctly however, so you only need to fix your Python installation:

  1. Go to https://www.python.org/downloads/ and download and install Python 2.7 (Python 3 will not work!)
  2. Locate your Python installation, e.g., c:\Python27\python.exe
  3. With the same user that will execute the maven build, run npm config set python c:\\Python27\\python.exe (replace with your actual location, of course, and note the double backslashes).

9. Copy and Adjust Configurations

  1. Copy the webofneeds conf-folder: cp -r conf conf.local if you haven't done that already.

  2. Change all instances of localhost in the configurations to your your IP (or computer name or domain) if you are not going to run node owner or matcher locally.

  3. In conf.local, edit matcher-service.properties, node.properties and owner.properties and change all instances of keystore/truststore locations to point to a folder where you will store generated keys and certificates (in ), e.g. I used /home/username/WoNKeystore/). Do not change the file name, just the path.

  4. Change the accompanying passwords to something at least 6-letter long.

10. Download/Install the latest Tomcat 9 server

sources:

11. Add Tomcat to Eclipse

  1. On Linux: You might need to add your user to the "tomcat" group (e.g. via sudo usermod -aG tomcat myusername). Check the permissions on your tomcat-installation/config-directory (e.g. /usr/share/tomcat/). Otherwise the integration process might throw an error that it couldn't read the configs.
  2. Add Server view: Window >> Show View >> Server. When the server appears there, double clicking it will allow to open the config-GUI.
  3. Create Server in Eclipse: File >> New >> Other >> Server
  4. Choose Tomcat 9, then press "next" (not "finish")
  5. Make sure you use a Java 8 JDK or JRE, not java 9, or tomcat will not start up and throw a JAXB-related exception.
  6. Add node and owner and click finish
    1. If you do not have the options to add the owner and node application to the tomcat (also accessible via Server >> [your tomcat server] >> Add and Remove), something went wrong.
      1. Make sure to run maven install at least once
        • right-click "webofneeds" in the project explorer >> Run As >> Maven install
        • or in the command-line: mvn install -P skip-tests
      2. Make sure the following project facets are active for the won-owner-webapp and won-node-webapp-projects (right-click project >> Properties >> search: "facets" >> activate facets):
        • Dynamic Web Module
        • Java
        • JavaScript
      3. Do an eclipse build (Project >> Build All)
      4. Maybe the import of the webofneeds maven project somehow did not work properly. Delete all imported projects (without deleting the sources), then import again (File >> Import... >> Maven >> Existing Maven Projects )
      5. Maybe you did not install eclipse for Java EE. Check Help >> About Eclipse. If it does not say 'Eclipse Java EE IDE for Web Developers.', the easiest is to download and install Eclipse for Java EE.

12. Update Connector-Statement in Tomcat's server.xml

Copy the SSL connector statement given below to <TOMCAT_FOLDER>/conf/server.xml as a child of the <Service name="Catalina">-node and change the password and key-folders there to values used in previous steps as well (e.g. "changeit", /home/username/WoNKeystore/t-cert.pem, /home/username/WoNKeystore/t-key.pem). If you already have a Connector-statement, modify it accordingly. You can also access the conf in Eclipse if you've already added Tomcat there via Project Explorer >> Server >> "Your Server" >> open server.xml.

<Service name="Catalina">
 ...
 <Connector
   SSLEnabled="true"
   compressibleMimeType="text/html, text/xml, text/plain, text/css, text/javascript, application/javascript, application/x-font-ttf, image/svg+xml, text/turtle, application/rdf+xml, application/x-turtle, text/rdf+n3, application/json, application/trig, application/ld+json, application/n-quads"
   compression="on"
   disableUploadTimeout="true"
   enableLookups="true"
   maxThreads="200"
   minSpareThreads="5"
   port="8443"
   protocol="org.apache.coyote.http11.Http11Nio2Protocol"
   scheme="https"
   secure="true">
     <SSLHostConfig
      certificateVerification="optionalNoCA"
      certificateVerificationDepth="2"
      trustManagerClassName="won.utils.tls.AcceptAllCertsTrustManager"
      protocols="all">
      <Certificate
        certificateKeystoreFile="/home/username/WoNKeystore/t-keystore.jks"
        certificateKeystorePassword="changeit"/>
   </SSLHostConfig>
   <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" /> 
 </Connector>

13. Edit the Server Configuration in Eclipse

DoubleClick the server in the "Server View" and select:

VM arguments

Open launch configuration >> (x)= Arguments >> VM arguments * append the following, replacing the placeholders <PROJECT_FOLDER> to the webofneeds project location and the <TOMCAT_FOLDER> to the installation directory of your tomcat:

-DWON_CONFIG_DIR="<PROJECT_FOLDER>\webofneeds\conf.local" -Dlogback.configurationFile="<PROJECT_FOLDER>\webofneeds\conf.local\logback.xml"

Add the JSTL jar to classpath

Add the JSTL jar to your tomcat's classpath libs (or tomcat cannot be run in Serve modules without publishing mode, see Server Options below):

  • Open Launch Configuration >> Classpath >> User Entries >> Add External JARs ...
  • Navigate to your maven repository (default location: $HOME/.m2/repository; if it's not there, look into $HOME/.m2/settings.xml)
  • find javax\servlet\jstl\1.2\jstl-1.2.jar
  • if you don't find it
  • build the whole project again via mvn install -Dmaven.test.skip=true or eclipse's maven integration (right-click on "webofneeds" >> Run As >> Maven Install).
  • try again

Add the required libs to the classpath

Like above, add all jars in webofneeds/target/required-libs. This should include won-utils-tls-x.y-jar.

Server Overview Options

In the Server overview (reachable via double-clicking the server in the servers-tab at the bottom):

  • select "Server Locations >> Use workspace metadata".
  • Server Options
    • Serve modules without publishing _(allows for instant effect of changes)*
    • Publish module contexts to separate XML files
    • Modules auto reload by default
    • Enable security
    • Enable tomcat logging
  • Publishing: Never publish automatically
  • Timeouts: i.e. 180 + 30
  • Ports: The ports should be shown for HTTP + SSL

Options to speed up Startup

Suppress unnecessary tag library (TLD) scans to speed up startup:

  • In the eclipse navigator view, open Servers >> Tomcat 9.0 _(your server config)*
  • edit catalina.properties
  • replace the value of the multi-line(!) property tomcat.util.scan.StandardJarScanFilter.jarsToSkip such that the line reads, tomcat.util.scan.StandardJarScanFilter.jarsToSkip=*.jar
  • replace the value of the multi-line(!) property tomcat.util.scan.StandardJarScanFilter.jarsToScan such that the line reads, tomcat.util.scan.StandardJarScanFilter.jarsToScan=jstl-1.2.jar

Allow both webapps (owner-webapp and node-webapp) to start simultaneously:

  • In the eclipse navigator view, open Servers >> Tomcat 8.0 _(your server config)*
  • edit server.xml
  • find the xml element <Host appBase="webapps" ... and add the xml attribute startStopThreads="2"

14. Generate cryptographic keys

NOTE: The following guide assumes you want to run all services on the same machine. If you deploy these on different machines, use the respective IPs and Paths as suited/desired.

  1. We use the Apache Portable Runtime (APR) in the security modules of the Web of Needs. Normally, APR is included in the Tomcat by default, but it could happen (e.g. when you use the Tomcat for Windows installer) that it's not included. To be sure that you have APR, check your <TOMCAT_FOLDER>/bin. A DLL file called tcnative-1.dll MUST be there (if you're on Windows). On Linux it exist occur globally e.g. in /usr/local/apr/lib/libtcnative-1.a

  2. In the console navigate to the folder for the keystore created in previous steps (e.g. /home/username/WoNKeystore/), adapt(!) and run the following lines:

    openssl req -x509 -newkey rsa:2048 -keyout t-key.pem -out t-cert.pem  -passout pass:changeit -days 365 -subj "/CN=myhost.mydomain.com"

openssl pkcs12 -export -out sometmpfile_deletme -passout pass:changeit -inkey t-key.pem -passin pass:changeit -in t-cert.pem

"$JAVA_HOME/bin/keytool(.exe)" -importkeystore -srckeystore sometmpfile_deletme -srcstoretype pkcs12 -destkeystore t-keystore.jks -deststoretype JKS -srcstorepass changeit  -deststorepass changeit

rm sometmpfile_deletme

    NOTE: the openssl commands can be executed in windows using cygwin or the git bash
    NOTE: If you're getting the error message Subject does not start with '/'., change last parameter to -subj "//CN=myhost.mydomain.com"

  3. The other key stores, and the trust stores are created and filled in automatically when the application is run (in the locations defined in step 4 with the passwords defined in step 5).

  4. For Oracle Java 8: Depending on your java-setup it might not be able to generate keys of a relevant length. In that case, you need to download and install the Java Cryptography Extension. There's a readme in the zip detailing its setup. At the time of this writing, this consists of copying the two jars into $JAVA_HOME/(jre/)lib/security. If you don't do this or the jars are in the wrong folder, you'll get an exception like java.security.InvalidKeyException: Illegal key size when trying to run the app. Check out step 3 here for a more detailed guide. In OpenJDK the unlimted cryptography policy should be enabled by default.

NOTE: If you're re-deploying the project and want to use different digital certificates than previously used (e.g. by generate new ones according to step 8, or use your server certificates certified by a CA): in addition to replacing corresponding server certificate files and the broker's key store, you have to also delete (or empty) all the trust-store files used by the applications. Also, if you have deleted or replaced the owner keystore, and your deployment uses something other than the in-memory database, you have to empty this database as well -- the stored data related to owner registration at node has to be deleted.

NOTE: If you're running docker containers:

Deploy sripts for building and running web of needs as docker containsers (see webofneeds/webofneeds/won-docker/deploy*.sh) include building and running server/broker certificate generation container gencert. It generates self-signed certificates for the server with the specified name (or IP) protected with the specified password if there are no certificates already present in the mounted volume. The generated key and certificate are in pem format (for Tomcat server) and java keystore format (for broker). If you already have server certificate for your server, and they are in the required format, you don't need to run the gencert. If you do want to generate and use self-signed certificate for your web of needs deployment, run it with changed parameters (server name and password), so that they correspond to your server name and to your desired password. Make sure that the same volume containing the certificate is mounted to the wonnode or owner container it is intended for. E.g. the location with the certificate of server.example.at should be mounted when owner application is run as docker container if the owner is being deployed at server.example.com. When using other than default file pathes and passwords for certificates, make sure that the same values are used in application properies files and in server.xml.

NOTE: Inspecting keystores using keytool: owner/node keystores are saved in bouncycastle's UBER format. keytool -list -v -providerclass org.bouncycastle.jce.provider.BouncyCastleProvider -storetype UBER -storepass <YOUR_KEYSTORE_PASSWORD> -keystore t-keystore.jks

15. Install the bouncycastle security provider and the trust manager

Locate the JRE you are using with eclipse (Window -> Preferences -> Java -> Installed JREs).

  • Navigate to the [JRE]/lib/security folder

  • look for the java.security file in your JDK's folder structure (for Oracle Java it should be in $JAVA_HOME/lib/security, in OpenJDK-12 it was in $JAVA_HOME/conf/security)

  • edit the file java.security

  • find the List of providers and their preference orders, which looks like this:

    security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
security.provider.3=sun.security.ec.SunEC
...

  • add this line (replace 11 with the last number in the list plus one)

    security.provider.11=org.bouncycastle.jce.provider.BouncyCastleProvider
  • For Java 8: the contents of [won-checkout-dir]/webofneeds/webofneeds/target/required-libs/ (which will be there after the first build) to the [JRE]/lib/ext/ folder
  • For Java >=9: add the contents of [won-checkout-dir]/webofneeds/webofneeds/target/required-libs/ to the front of your class-path (support for lib/ext has been dropped with java 9). If you're using Eclipse for the build, you can do so via "[Run Button] >> Run Configurations >> [tomcat run config name] >> Classpath >> select 'Bootstrap Entries' >> Add JARs"

(if you miss this step, you'll see BC exceptions when running the owner/node)

16. Start server

Select Server in the "Servers"-Tab at the bottom and click play (or right-click the server and press "play" in the context-menu)

Troubleshooting

Tomcat complains about Missing Keystore

SEVERE [main] org.apache.tomcat.util.net.jsse.JSSESocketFactory.getStore Failed to load keystore type JKS with path C:\Users\[username]/.keystore due to C:\Users\[username]\.keystore
 java.io.FileNotFoundException: C:\Users\[username]\.keystore

Reason: we use tomcat APR. This means that in conf/server.xml, the following line must be present:

<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />

and the apache tomcat native library needs to be installed. On Windows, this means you find the file tcnative-1.dll in tomcats lib folder, for more information, see http://tomcat.apache.org/native-doc/.

On linux you might need to build it for yourself, see "Ubuntu Installation Notes - Building TCNative](./ubuntu-installation-notes.md#building-tcnative)

Tomcat start fails with ClassNotFoundException: java.lang.ClassNotFoundException

Error configuring application listener of class [org.springframework.web.context.ContextLoaderListener]
java.lang.ClassNotFoundException: org.springframework.web.context.ContextLoaderListener

Reason: The Maven Dependencies are not defined in the Web Deployment Assembly of the Web Application project.

Solution:

  1. Right click on won-owner-webapp project >> Properties >> Deployment Assembly >> Add... >> Java Build Path Entries >> Maven Dependencies.
  2. Project >> Clean... >> select won-owner-webapp >> select Build only the selected projects >> Clean.
  3. Do a maven install
  4. Right click on server >> Publish >> Start

Maven build fails with NoClassDefFoundException: java/sql/SQLException

Turns out that you need to have a JDK installed. Download a recent JDK and tell eclipse to use that one by default: Window >> Preferences >> Java >> Installed JREs. For me, Jdk11 did not work, but Jdk8u202 did.

Out of memory error

If you run your project and encounter a "Out of memory" error you should probably add this to your run configuration: -XX:MaxPermSize=128M. This should only be necessary if you use JDK 1.8.

icu4j: Invalid byte tag in constant pool

If you get a compile error like below you can just ignore it. There is a corrupt .class file in the maven dependencies and will go away with future updates and everything is OK untill you use it, but you won't. This will only affect Chinese speaking users since it is the LocaleElements_zh__PINYIN.class file for Pinyin.

Another possibility is to find the icu4j .jar file and delete the LocaleElements_zh__PINYIN.class file. This is a quick and dirty hack, but it works.

Apr 16, 2013 2:07:59 PM org.apache.catalina.startup.ContextConfig processAnnotationsJar
SEVERE: Unable to process Jar entry [com/ibm/icu/impl/data/LocaleElements_zh__PINYIN.class] from Jar [jar:file:/C:/DATA/atus/Code/webofneeds/webofneeds/won-node-webapp/target/won/WEB-INF/lib/icu4j-2.6.1.jar!/] for annotations
org.apache.tomcat.util.bcel.classfile.ClassFormatException: Invalid byte tag in constant pool: 60
at org.apache.tomcat.util.bcel.classfile.Constant.readConstant(Constant.java:133)
at org.apache.tomcat.util.bcel.classfile.ConstantPool.<init>(ConstantPool.java:60)

More info about this error can be found at:

won.protocol.exception.RDFStorageException: Could not create File!

This means that Tomcat could not access the temp directory where it stores the .ttl files. This is either the TMP global variable or the %tomcat_dir%/tmp directory. The problem is that if you're on Windows and installed Tomcat in C:/Program Files/. Unless you are an admin you will not have access to any of these.

You can either grant your user access to one of these directories or change the directory in won-node/src/main/resources.localhost/ by changing the rdf.file.path.

Check the error message for the actual directory in question. Expect something like

Caused by: java.io.FileNotFoundException: C:\Program Files\apache-tomcat-7.0.35\temp\1.ttl (Access denied)

Port Bind Problem - Failed to initialize end point associated with ProtocolHandler

org.apache.coyote.AbstractProtocol.init Failed to initialize end point associated with ProtocolHandler ["http-apr-8443"]

This problem or other similar errors that can be referred as "port bind problems" are causd by setting the port that is used by WoN-node or WoN-webapp in the Tomcat Server Settings of IntelliJ. Leave the field "HTTP port" in the "Run/Debug Configurations" free (the default is 8080 and does not cause any problem).

maven dies in won-owner-webapp during 'clean' task

stating that 'node' was not found. solution: run mvn install before mvn clean

java.security.NoSuchProviderException: no such provider: BC

Make sure you've gone through the steps in "Install the bouncycastle security provider and the trust manager".

If this error still occurs, it could happen that Tomcat can't find or access the the bc .jar files during startup. Below is a collection of actions that may fix the problem and places to copy the .jar files into. You may want to try them both separately and combined to find a setup that works for you.

You should be able to find both bcpkix-jdk15on-1.64.jar and bcprov-jdk15on-1.64.jar in your maven directory (default location is: C:/Users/[user name]/.m2/repository/org/bouncycastle/). If not, build the whole project with mvn install -Dmaven.test.skip=true and check again.

  • in Eclipse, edit the server launch configuration properties (accessible via "Run As...") and add both .jar files as External JARs to User Entries (suggested in: Building, Step "Add the JSTL jar to classpath")
  • copy both bcpkix-jdk15on-1.64.jar and bcprov-jdk15on-1.64.jar to %tomcat_dir%/lib/ (suggested in: Crytpographic Keys and Certificates, Step 11)
  • copy both bcpkix-jdk15on-1.64.jar and bcprov-jdk15on-1.64.jar to C:/Program Files/Java/[jre dir]/lib/ext/ and install the bouncy castle security provider (suggested in: issue#1393)
  • In the Tomcat server's server.xml, find the xml element <Host appBase="webapps" ... and add the xml attribute startStopThreads="2"
    • NOTE: This only has an effect if two or more webapps, e.g. a node and an owner, are started on the server.

Exception in Owner-Webapp log: PKIX path building failed

This happens when the webapp uses a different public key than the activemq http endpoint.

We have seen this happen in two different ways:

  1. The keys in the pem files and in the jks files are identical, but they were changed since the applications were started - most likely by a letsencrypt certificate renewal.
  2. There are different keys in the pem files than in the jks and pfx files

If you suspect 1 might be the case, just restart the docker containers. If the problem goes away, that's what it was.

In case 2, probably the certificate renewal updated the pem files but did not update the jks and pfx files. The consequence is that the node webapp uses the new key (as nginx loads the pem file) and the activemq server uses the old key (as it loads the jks file).

To check if this is the problem: list the keys in the jks/pfx file (using keytool -list -v -keystore t-keystore.pfx) and compare them to the key information available for, eg. https://{your-won-node}/won/resource in the browser or some other http client. If the keys are the same, this is not the problem.

To fix this: overwrite the jks and pfx files by exporting the key from the pem file. This can be done by executing the openssland keytool commands in the letsencrypt's container script certificate-request-and-renew.sh