Contributing to AdoptOpenJDK/Infrastructure

Introduction

I recently joined the AdoptOpenJDK community, and chose to focus on helping to maintain the build and test machines and the automation used to set up and maintain them.

It’s something I’ve been meaning to do for a while, but for one reason or another I’ve not got round to it. I think one of those reasons was that I thought it would be difficult, and involve me having to talk to lots of people, all giving me slightly different advice.

I found that it wasn’t difficult at all, and the people I spoke with were all very helpful.

What I did

I’ve described the steps I took to get onboard in the AdoptOpenJDK Infrastructure wiki here. I put them there to make it easier for anyone to update should any processes or tools change.

Get involved

Many of the steps in my instructions are common to all of the AdoptOpenJDK projects. If you go to https://github.com/AdoptOpenJDK you can see all 49 repositories. Please get involved if you think you’ve got something to offer any of them.

 

Adding zOS to our Jenkins Build Farm

Introduction

Adding zOS to a Jenkins server as an ssh agent can be a difficult task, that’s why I’ve documented the steps that we were required follow to add our zOS machines to Jenkins.

Adding Java to zOS

All Jenkins build agents require Java to be on the machine to run the Slave agent. Normally an OpenJDK binary would be recommended to run the slave agent but there is no official zOS OpenJDK binary so I would recommend fetching IBM’s Java from here.

Adding the Machine to Jenkins

Head over to https://<jenkins-server>/computer/new, specify a node name and tick permanent agent.

Jenkins Screenshot 1

Once you have reached the machine configuration page specify the Launch method as Launch slave agents via SSH. Set the host IP Address, Credentials and Host Key Verification Strategy and then click advanced.

Jenkins Screenshot 2

In the advanced page, set the JavaPath as the path to the Java binary that you downloaded earlier and then add the following to JVM Options:

<br>-Dfile.encoding=ISO8859-1<br>

This will allow Jenkins to understand the Ebcdic output from Java on the machine.

You should now be able to successfully launch the Jenkins Java agent on your zOS machines.

SSH Key Management with KeyBox

Any project with more than a handful of machines will appreciate how much work is required to maintain them and ensure that access is kept to the minimum number of people. When we started creating our build farm we quickly realised how time-consuming this could be and so we wanted to find a neat solution to save us some time. One of the biggest issues that we found was giving people temporary access to machines. We would often find that the access was never revoked and this therefore became an administrators nightmare.

Whilst searching on the internet I came across an open-source project named KeyBox. KeyBox is a web-based SSH console that centrally manages administrative access to systems. It is super easy to setup and we decided to run it on one of our machines bundled with Jetty. After downloading a tar.gz file, there are just a handful of commands to run before you have a fully functioning KeyBox server. One of the other really great features is that you can install FreeOTP or Google Authenticator to enable two-factor authentication with Android or iOS.

Once you have a running instance of KeyBox you can start to change your configuration to suit what you need. We decided that we wanted to use our own existing public ssh keys rather than having KeyBox generate one, so we had to change the KeyBoxConfig.properties file. This file can be found at KeyBox-jetty/jetty/keybox/WEB-INF/classes. Simply edit this file with your favourite editor and change the line:

#set to true to generate keys when added/managed by users and enforce strong passphrases set to false to allow users to set their own public key
forceUserKeyGeneration=true

to be:

#set to true to generate keys when added/managed by users and enforce strong passphrases set to false to allow users to set their own public key
forceUserKeyGeneration=false

Then simply save your config and restart KeyBox using ./startKeyBox.sh

Adding a Machine

The first thing I did when I got KeyBox was add some of our machines. It's really simple to add machines, you just need to click 'Systems', 'Add System' and then fill in a display name, system user, and host (leave the port and Authorized Keys fields as default).

Once you click 'Submit', KeyBox will make an ssh connection to the machine. You will be prompted to enter the password for the user but you should only need to do this once. This is where the magic now happens…. KeyBox will erase all of your authorized_keys and add its own public key. One thing to consider is that when you add a new system, you are only adding it for that one single user so you may want to add the system multiple times (in our case we have a sudo user on every machine as well as a jenkins user with no administrative permissions.

Assigning systems to profiles

Once you've added some machines to KeyBox the next obvious thing to do is start assigning machines to profiles. The idea behind profiles is that you can create pools of machine access that you can then assign ssh keys to. For example in our case, we have a jenkins profile which contains all of the systems with jenkins user access. You may also want to create a Mac profile or a Linux profile to allow users access to certain operating systems only. Creating profiles is simple, you just navigate to the profiles page, click 'Add Profile' and then set and name and optional description. Once you have done that you can click 'Assign Systems' and tick which machines you want to assign to that profile.

Adding SSH Keys

This is the last step for us, you have successfully added all of your machines and assigned access profiles to them, you now want to give people access. As I mentioned above, we wanted users to be able to send me their existing public key, and for that key to be added to all machines that they need rather than generating keys and sending them back to the user. This way seemed more logical to me as it prevents users ending up with several keys to get lost on their system.
To add a users key, simple click 'Add/Remove Keys', copy in their public key, specify which profile to assign the key to and then click 'Submit'. This is a great way to handle key management as it means that the administrator has one simple view to see all machine access by every user.

Other features

KeyBox has several other great features such as two-factor authentication, web based terminal sessions and group code execution but I haven't tested those features enough to write about them yet. All in all, KeyBox is a great tool, that I would recommend to any open-source project that needs better control when it comes to ssh key management.

Find our more here.

Creating an Open Source build farm

When it comes to any community driven open source project, funding is always the limiting factor. There is no cheap way to get machines hosted for you in the cloud so it’s essential that you approach as many companies as you can and try and build up sponsorship deals. We currently have over 30 build machines in our Jenkins farm as well as several other machines hosting our other key services such as the API, Keybox and our Semaphore Ansible box.

How to approach companies

The first thing that you must consider when approaching companies is what can you do for them? Can you offer them promotion in exchange? Or perhaps you can help them by beta testing some of their new systems? When it comes to contacting the company I tend to use their existing contact email from the website. You don’t need to write a very long email but it’s worthwhile tailoring each email for the company that you’re writing to. For example don’t write to a Mac hosting company requesting some “machines”, ask for some 4cpu, 16gb Mac Mini’s. Make sure that you explain what you do and why you need the machines, remember to include how you can help them in exchange. One of the main things to remember when approaching companies asking for sponsorship is that a lot of the time, they might not have an existing system in place to provide sponsored or “free” machines. It can therefore take a long time from your initial point of contact before you get a final decision.

Sponsorship options

Every company will interpret sponsorship in a different way. Some will just hand over machines, others will give you a discount, and others may give you a prepaid hosting account. All of these options can have a different monetary value. You may have $1500 from one company and $20 from another. It’s also worth considering the cost of what your sponsor is giving you, for example AIX hosting will cost a lot more than standard x86 Linux hosting. You may therefore find that some companies offer you less than others.

Success! You have some machines…

If you are lucky enough to have been offered some machines then the first thing to remember is that the company has probably had to pull quite a few strings to make this happen. Make sure that you express your gratitude to the company and assure them that the machines will be useful. Make sure that you put the machines to use as quickly as possible otherwise it looks like you didn’t really need them in the first place, and you don’t want the company to question their sponsorship.

If you aren’t so lucky…

Don’t be disappointed if the answer is no, there are many hosting companies out there and there will almost certainly be a handful that are interested in your organisation. For a lot of companies, sponsorship just isn’t on the table. It’s also worth asking them if they can suggest any other companies that may be interested.

Conclusion

We’ve been overwhelmed by the generosity of our sponsors. I’m really proud that we have built up so many connections with leading companies in the hosting industry. Check out our sponsors page here to see all of our current sponsors. Remember to keep your sponsors up to date with progress as this is what ultimately leads to a long term sponsorship.

AIX is here!

A couple of weeks ago we had a pair of 10 cpu, 32gb ram AIX machines setup for us at the OSU Open Source Lab in Oregon. We have been playing around with the configuration on them for the last few weeks but I think that we’ve got our build recipe right now. Go ahead and grab our binaries from here.

Please not that these binaries are currently untested and are not recommended for production use.