Playing with Test Microservices

Recently, I had the good fortune to speak (fondly!) about some excellent open-source projects that I participate in…  AdoptOpenJDK, Eclipse OpenJ9 and Eclipse OMR.  Across those three projects, we are trying to simplify the activity of testing.  One of the great side-effects of simpler, easier testing is that it frees up time to do some fun work around building microservices for test.  We hope that we can eventually bring the best of these microservices into the open also, to help all open-source endeavours be even better at the activities of test.

If you are interested in hearing more about microservices for test, please give a listen to “Cloud-based Test Microservices“, a presentation which I gave at the Eurostar conference in Copenhagen this year.

 

OpenJ9 builds available here!

250,000 hits in 24 hours!

What a crazy last week it has been for everyone at AdoptOpenJDK. We are very excited to have begun adding OpenJ9 builds to our website! The interest has been overwhelming and it was incredible to see our website reach well over 250,000 hits after a small thread on Reddit quickly became much more! The picture below from our Cloudflare analytics tool shows very clearly when the post was added to Reddit and the views kept pouring in for the rest of the day.

Platform Support

We currently only have builds for x86, s390x and ppc64le Linux but we plan to add Windows, macOS and many more as soon as the OpenJ9 team is ready. The easiest way to download them is at adoptopenjdk.net.

To read more about the advantages of using OpenJ9 over Hotspot read the OpenJ9 FAQ here.

New contribution of test cases to AdoptOpenJDK

We have contributed additional tests and a test framework the tests depend on, to the AdoptOpenJDK project in these repositories: openjdk-systemtest and stf.

The tests are longer running tests which cannot be automated using standard unit test frameworks (such as JUnit or TestNG). They fall under the broad category of ‘system tests’ – tests which attempt to simulate running production workloads and specific user scenarios.

The tests are mostly load tests which run a collection of java methods (typically discrete test case methods) for either a set number of invocations, or a period of time, using one or more threads. Although the tests can (and do) find functional issues executing the individual methods in the load, that is not the main goal of the testing. Running a workload for a period of time is often the most effective way to identify defects in the java virtual machine such as the garbage collector and the dynamic java compiler.  See the load test tool documentation for more details of these tests.

Other tests are more multi-process, multi-step in nature. For example, tests which use the Java Debugging Interface (JDI) to examine a test program running in a second JVM, and tests which perform remote operations on a second JVM running a workload, such as attaching a Java transformer agent.  See here for more details of these tests.

You can download and execute the tests locally – follow the instructions at openjdk-systemtest. You will also find additional documentation about the tests and the test tooling in the repositories.

 

We’re off to a great start

AdoptOpenJDK has certainly come a long way over the last few months!

We have a clean, bright website to distribute binaries of OpenJDK, with exciting plans for how it can be improved in the future.

Behind the website “shop front” there is a large continuous integration build and test farm, covering multiple operating system and hardware combinations.  The farm is put to good use compiling the OpenJDK code and running it through a suite of tests before publishing it.

All of our code is out there in Github for you to study, so you know exactly what those binaries contain and how they were built.  We believe in open build and open testing of the open source code!

We started with OpenJDK version 8 as the latest stable code stream.  Now that the OpenJDK project is close to declaring the OpenJDK version 9 final, we have started building and testing that too, and expect to have tested binaries available from the AdoptOpenJDK website and via our API simultaneously.

We are also happy to see the proposal for Eclipse OpenJ9 as another main stream open source Java Virtual Machine.  Once that code is available we will take a look with a view to building and testing OpenJDK with Eclipse OpenJ9 binaries too.

As an open build, test and distribution project we aim to be the “go to” location for high quality OpenJDK-based runtimes; and we are working hard to earn that reputation.  If you are maintaining an OpenJDK derived runtime, drop us a note.

There are many interesting and challenging tasks ahead.  Everyone is welcome to participate at whatever level suits you best.  Take a look at some of the work we know we need to do, or join us on Slack to talk about your new ideas.

We are grateful to the sponsors who recognize the value of this project and have generously provided services and resources to make it successful.

The next few months are going to be equally exciting!  Join in.

Adding zOS to our Jenkins Build Farm

Introduction

Adding zOS to a Jenkins server as an ssh agent can be a difficult task, that’s why I’ve documented the steps that we were required follow to add our zOS machines to Jenkins.

Adding Java to zOS

All Jenkins build agents require Java to be on the machine to run the Slave agent. Normally an OpenJDK binary would be recommended to run the slave agent but there is no official zOS OpenJDK binary so I would recommend fetching IBM’s Java from here.

Adding the Machine to Jenkins

Head over to https://<jenkins-server>/computer/new, specify a node name and tick permanent agent.

Jenkins Screenshot 1

Once you have reached the machine configuration page specify the Launch method as Launch slave agents via SSH. Set the host IP Address, Credentials and Host Key Verification Strategy and then click advanced.

Jenkins Screenshot 2

In the advanced page, set the JavaPath as the path to the Java binary that you downloaded earlier and then add the following to JVM Options:

<br>-Dfile.encoding=ISO8859-1<br>

This will allow Jenkins to understand the Ebcdic output from Java on the machine.

You should now be able to successfully launch the Jenkins Java agent on your zOS machines.

GitHub Pages with Cloudflare

GitHub Pages offers the ability to host static websites from a GitHub Repository and it can be linked to CloudFlare using a custom domain name. Though it does not support SSL on custom domains, we can use CloudFlare Universal SSL to allow our users to access the website over SSL.

Only static HTML?

Even though GitHub Pages only has the ability to host static content it isn’t such a big cutback.This means that the web server only needs to deliver a static HTML to the end user, resulting in performance benefits.Furthermore, using the git repository we can track site changes and thus having a better control when shared coding.

Creating a GitHub Repository

Creating a GitHub Repository that contains the HTML files.

Website Repository

Enabling GitHub Pages

Once we have created our GitHub Repository , we need to enable GitHub Pages in Settings and specify a branch that will built the website, the default is the master branch.If we use a custom domain , we change the default domain to point to our custom domain.

Setting up DNS

After having registered the custom domain and added it to CloudFlare.

Custom Domain CloudFlare

We need to setup the CNAME file, in our GitHub Repository, which declares the hostname to accept traffic from.

Configuring SSL

GitHub Pages doesn’t yet support SSL for custom domains and thus ruling out HTTP/2. How can we get past it?
CloudFlare’s Universal SSL option give us the ability to provide a signed SSL certificate to site visitors thus gaining the performance benefits of HTTP/2. To set it up, we set the SSL mode to FULL in Cloudflare.

To enforce more restrictions we can add a page rules. For example, a page rule to enforce HTTPS.

 

Testing Java: Let Me Count the Ways

For years now, I have been testing Java and if there is a single statement to make about that activity, it is that there are many, many, many ways to test a Java Virtual Machine (JVM).

From code reviews and static analysis, to unit and functional tests, through 3rd party application tests and various other large system tests, to stress/load/endurance tests and performance benchmarks, we have a giant set of tests, tools and test frameworks at our disposal.  Even the opcode testing in the Eclipse OMR project helps to test a JVM.  From those low-level tests, all the way up to running some Derby or solr/Lucene community tests, or Acme Air benchmark there are many ways to reveal defects.  If we can find them, we can fix them… (almost always a true statement).

One common request from developers is “make it easier for me to test”.  Over the last year, we have been working on that very request.  Recently, I’ve had the good fortune to become involved in the AdoptOpenJDK project.  Through that project, we have delivered a lightweight wrapper to loosely tie together the various tests and test frameworks that we use to test Java.

We currently run the set of regression tests from OpenJDK (nearly 6000 test cases).  Very soon, we will be enabling more functional and system-level tests at that project.

My goal with that project is to make it super easy to run, add, edit, exclude, triage, rerun and report on tests.  To achieve that goal, we should:

  • create common ways of working with tests (even if they use different frameworks, or focus on different layers in the software stack)
  • limit test code/infrastructure bloat
  • choose open-source tools and frameworks
  • keep technical ego in check to restrict needless complexity for little or no gain in functionality

There is a lot of work ahead, but so far, its been fun and challenging.  If you are interested in helping out on this grand adventure, please check out the project to see how to get involved at AdoptOpenJDK.

SSH Key Management with KeyBox

Any project with more than a handful of machines will appreciate how much work is required to maintain them and ensure that access is kept to the minimum number of people. When we started creating our build farm we quickly realised how time-consuming this could be and so we wanted to find a neat solution to save us some time. One of the biggest issues that we found was giving people temporary access to machines. We would often find that the access was never revoked and this therefore became an administrators nightmare.

Whilst searching on the internet I came across an open-source project named KeyBox. KeyBox is a web-based SSH console that centrally manages administrative access to systems. It is super easy to setup and we decided to run it on one of our machines bundled with Jetty. After downloading a tar.gz file, there are just a handful of commands to run before you have a fully functioning KeyBox server. One of the other really great features is that you can install FreeOTP or Google Authenticator to enable two-factor authentication with Android or iOS.

Once you have a running instance of KeyBox you can start to change your configuration to suit what you need. We decided that we wanted to use our own existing public ssh keys rather than having KeyBox generate one, so we had to change the KeyBoxConfig.properties file. This file can be found at KeyBox-jetty/jetty/keybox/WEB-INF/classes. Simply edit this file with your favourite editor and change the line:

#set to true to generate keys when added/managed by users and enforce strong passphrases set to false to allow users to set their own public key
forceUserKeyGeneration=true

to be:

#set to true to generate keys when added/managed by users and enforce strong passphrases set to false to allow users to set their own public key
forceUserKeyGeneration=false

Then simply save your config and restart KeyBox using ./startKeyBox.sh

Adding a Machine

The first thing I did when I got KeyBox was add some of our machines. It's really simple to add machines, you just need to click 'Systems', 'Add System' and then fill in a display name, system user, and host (leave the port and Authorized Keys fields as default).

Once you click 'Submit', KeyBox will make an ssh connection to the machine. You will be prompted to enter the password for the user but you should only need to do this once. This is where the magic now happens…. KeyBox will erase all of your authorized_keys and add its own public key. One thing to consider is that when you add a new system, you are only adding it for that one single user so you may want to add the system multiple times (in our case we have a sudo user on every machine as well as a jenkins user with no administrative permissions.

Assigning systems to profiles

Once you've added some machines to KeyBox the next obvious thing to do is start assigning machines to profiles. The idea behind profiles is that you can create pools of machine access that you can then assign ssh keys to. For example in our case, we have a jenkins profile which contains all of the systems with jenkins user access. You may also want to create a Mac profile or a Linux profile to allow users access to certain operating systems only. Creating profiles is simple, you just navigate to the profiles page, click 'Add Profile' and then set and name and optional description. Once you have done that you can click 'Assign Systems' and tick which machines you want to assign to that profile.

Adding SSH Keys

This is the last step for us, you have successfully added all of your machines and assigned access profiles to them, you now want to give people access. As I mentioned above, we wanted users to be able to send me their existing public key, and for that key to be added to all machines that they need rather than generating keys and sending them back to the user. This way seemed more logical to me as it prevents users ending up with several keys to get lost on their system.
To add a users key, simple click 'Add/Remove Keys', copy in their public key, specify which profile to assign the key to and then click 'Submit'. This is a great way to handle key management as it means that the administrator has one simple view to see all machine access by every user.

Other features

KeyBox has several other great features such as two-factor authentication, web based terminal sessions and group code execution but I haven't tested those features enough to write about them yet. All in all, KeyBox is a great tool, that I would recommend to any open-source project that needs better control when it comes to ssh key management.

Find our more here.

Creating an Open Source build farm

When it comes to any community driven open source project, funding is always the limiting factor. There is no cheap way to get machines hosted for you in the cloud so it’s essential that you approach as many companies as you can and try and build up sponsorship deals. We currently have over 30 build machines in our Jenkins farm as well as several other machines hosting our other key services such as the API, Keybox and our Semaphore Ansible box.

How to approach companies

The first thing that you must consider when approaching companies is what can you do for them? Can you offer them promotion in exchange? Or perhaps you can help them by beta testing some of their new systems? When it comes to contacting the company I tend to use their existing contact email from the website. You don’t need to write a very long email but it’s worthwhile tailoring each email for the company that you’re writing to. For example don’t write to a Mac hosting company requesting some “machines”, ask for some 4cpu, 16gb Mac Mini’s. Make sure that you explain what you do and why you need the machines, remember to include how you can help them in exchange. One of the main things to remember when approaching companies asking for sponsorship is that a lot of the time, they might not have an existing system in place to provide sponsored or “free” machines. It can therefore take a long time from your initial point of contact before you get a final decision.

Sponsorship options

Every company will interpret sponsorship in a different way. Some will just hand over machines, others will give you a discount, and others may give you a prepaid hosting account. All of these options can have a different monetary value. You may have $1500 from one company and $20 from another. It’s also worth considering the cost of what your sponsor is giving you, for example AIX hosting will cost a lot more than standard x86 Linux hosting. You may therefore find that some companies offer you less than others.

Success! You have some machines…

If you are lucky enough to have been offered some machines then the first thing to remember is that the company has probably had to pull quite a few strings to make this happen. Make sure that you express your gratitude to the company and assure them that the machines will be useful. Make sure that you put the machines to use as quickly as possible otherwise it looks like you didn’t really need them in the first place, and you don’t want the company to question their sponsorship.

If you aren’t so lucky…

Don’t be disappointed if the answer is no, there are many hosting companies out there and there will almost certainly be a handful that are interested in your organisation. For a lot of companies, sponsorship just isn’t on the table. It’s also worth asking them if they can suggest any other companies that may be interested.

Conclusion

We’ve been overwhelmed by the generosity of our sponsors. I’m really proud that we have built up so many connections with leading companies in the hosting industry. Check out our sponsors page here to see all of our current sponsors. Remember to keep your sponsors up to date with progress as this is what ultimately leads to a long term sponsorship.

How to build OpenJDK…

Building OpenJDK with our makejdk-any-platform.sh script is really simple…

git clone https://github.com/AdoptOpenJDK/openjdk-build.git

Docker Build:

./makejdk-any-platform.sh

Optionally specify --destination <path> if you want the binary to be copied to your machine.

Local Build:

./makejdk-any-platform.sh --source $PWD --destination $PWD/OpenJDK.tar.gz

Troubleshooting:

If you have any issues build OpenJDK then you can check all of the command line flags using:

./makejdk-any-platform.sh --help

It’s also worth checking here for the latest issues if you are having problems.