Jenkins Auto-Configuration of GitHub WebHooks to Initiate Build Jobs

Wow. What a mouthful of a title.

It turns out that Jenkins has the capability to make calls to GitHub and set up WebHooks. A GitHub WebHook is a thing that can be set up to issue an HTTP(S) call back to Jenkins whenever things happen to a particular GitHub repo. In particular, things like a push to GitHub or a merge of a Pull Request will cause GitHub to notify Jenkins. What we want is for Jenkins to then pull the new code and do a build, running tests, and all that.

I am using GitHub Enterprise. That means is is an independent GitHub server (not the one on the internet at github.com) and it is running in my employer's internal network. This GitHub instance is only available to us when at work or when connected to work via VPN.

In Jenkins, I am using Cloudbees Enterprise Jenkins (the version with an Operations Center and one or more controlled Client Master instances). But this should work with the open source version of Jenkins where there is only a Master. The reason its the same is that in Cloudbees Enterprise Jenkins the Masters work with GitHub and the Operations Center does not. It means you have to configure all your masters separately.

Configure to Auto-Manage GitHub Hooks

For each master, go to the main dashboard page.

• Manage Jenkins > Configure System > GitHub > Add GitHub Server > GitHub Server (Note: I have two places to enter a GitHub server on this page. We are using the one that
• GitHub > GitHub Server > API URL:  something like https://ghub.mycompany.com/api/v3
• Click Advanced (The one after the “Add GitHub Server” button)
• Additional Actions > Manage additional GitHub actions > Convert login and password to token
• GitHub API URL (again) : https://aaghub.qcorpaa.aa.com/api/v3
• Select radio button to enter it from login and password
• Manage Hooks: Click to select
• Enter a GitHub account id and password. The account should have permission to Admin any repos that you will set up to be automatically built when the content changes. (Perhaps less permissions than that are needed. Dunno.)
• Click Create token credentials to generate
• Make note of the generated credential’s name (it’s a sequence of numbers)
• Save
• Manage Jenkins > Configure System > GitHub Servers: Comes back to same screen
• GitHub Server > Credentials: Select the newly created credential (with the long name)
• Save

Now any jobs that are set to build on a change from GitHub (and that have a GitHub repo URL available on the config page) will have Jenkins configure GitHub to notify whenever that repo changes. (Note: This will not apply for GitHub Organization job types or Multi-Branch Build job types as they extend from a different subclass that does not have that code in it yet.)

Now we need to figure out if its working:

Go to the main dashboard page on the master OR click down from there to a folder where you can add a build job.

• New Item
• Name: TestJob1
• Click Freestyle Job
• Click OK

It will go the page where you add a freestyle job.

• Source code management > select GitHub
• Repository URL: enter one
• Credentials: Select the right one (I'm assuming you know how to access a GitHub repo code)
• Build Triggers > Select Build when a changed to pushed to GitHub
• For now select Build > Execute Shell (or Windows Batch Command if appropriate)
• In the shell script box, I put "ls -al" (or "dir", respectively).
• Save

Now to the the GetHub Web UI, log in using an account that has admin permissions on the repo you are using.

• Move through the web pages to the main page for the repo
• Click "Settings" at the right side of the top line of options
• Click "Hooks and Services" in the left, vertical menu
• Verify that there is a webhook pointing to your Jenkins server. It will use the URL that you configures as the URL of the server in "manage jenkins > configure system".

If so, you are golden. If not, you have a bunch of debugging to do. I don't know all the ways it can go wrong.

Maybe the Jenkins server cannot connect to the GitHub server over the internet. Go to a command line on the Jenkins server and ping or telnet to the GitHub server and see if it connects. 

Maybe the GitHub server cannot connect to the Jenkins server. Go to a command line on the GitHub server and see if you can ping or telnet to the Jenkins server. This may not be permitted. 

You can also test this by entering the web hook by hand in the GitHub repo and testing it. Note that if you leave the one you entered by hand, you will never know if the automatic method is going to work the next time you create a build job needing to auto-build from GitHub.

Maybe you go to the Manage Jenkins page and a red box is at the top telling you that some GitHub WebHooks didn't get created. Click to view this and you might get a clue as to what is wrong. If the credentials you selected when configuring the system under Manage Jenkins are invalid or do not have admin permissions in GitHub, this sort of error can happen.

That page you just viewed may have information about how to see the log messages relating to GitHub web hooks. Set up a logging context with these three items:

1. com.cloudbees.jenkins.GitHubPushTrigger
2. com.cloudbees.jenkins.GitHubWebHook
3. org.jenkinsci.plugins.github

Set the log level to "ALL" for these three.

Then go configure the test job again and save it again. This seems to retry setting of hooks sometimes. (But if you have messages about the hooks not being created, you will need to "ignore" each and then "disignore" each again.) I'm not really sure what it is that makes Jenkins try to create the webhook again if it doesn't work the first time. Sometimes I would just go build the test job and see if the hook got set up. Someone should document this.

Cloudbees Enterprise Jenkins Timezone When Machine is UTC

I did this on a Linux Red Hat Enterprise 7 machine. The problem I was having had to do with the default time zone on Linux was UTC. When I would do "time" from a command line, it would show UTC time. In my case, it was 6 hours later than the time on my phone.

Jenkins uses time in several ways and you have to make more than one change to make all of them use the correct time zone.

Note that there doesn't seem to be a way to have Jenkins figure out where the user sitting in front of a browser viewing a Jenkins screen may be located. If the server is in one time zone and the user is in another time zone, I'm hoping to have the time displayed equal the server's time zone.

Here are the places I found to check whether Jenkins is doing the time zone the way I want:

1. Go into a build job and look in the lower left margin. The time of the builds is shown there. It should be the time zone you want.
2. Go to a pipeline build job and look at the time displayed beside the flow diagrams for the builds. Again, this should be your desired time zone.
3. Go to the CJOC > Manage Jenkins > System Logs > All Jenkins Logs and notice the time of the last logged event. It should be in the desired time zone. If your server has been idle for a while, it might not contain a recent log message. Take this into account.
4. Repeat the same thing for the Client Masters. Each of them should display correct times in the log messages.
5. Go to any build job that runs at a particular time of day. On the Configure screen, there is an item for build periodically > schedule that shows one or more "cron" type strings. The 2nd column is for the hour. The message below the schedule box should show the correct time zones for the times the job was and will be run.

• A cron string "0 1 * * *" will run at 1 am every day. Check that's what it says below the schedule box.
• A cron string "H H(9-17) * * *" will run every day once per hour between 9 AM and 6 PM. The H means Jenkins gets to pick the minute value randomly.
• A cron string "@midnight" will run every night between midnight and 6 am.

1. Similarly, the times the jobs run (after they run and the time of running is in the left margin at the bottom on the build job page) should be correct in the desired time zone. You may also be able to check the time of build by looking at the dates on built artifacts--stuff like build log files and jar files and such.

1st Change

Create a file, $JENKINS_HOME/init.groovy.d/my-timezone.groovy (which Jenkins will run upon startup). Do this on the CJOC (Operations Center) machine and on any Client Masters.

System.setProperty('org.apache.commons.jelly.tags.fmt.timeZone', 'America/Chicago')

Note that you will use one of the Java timezone strings. I'm in the Central time zone of the United States. (See http://stackoverflow.com/questions/1694885/timezones-in-java)

This makes it so that Jenkins will use the defined time zone when it is formatting certain strings to contain the time of day of some event (like a log message or a build time). It sort of depends on which bit of code is doing the formatting.

2nd Change

Then you need to add a startup string to the sysconfig file for jenkins. (How do you find it? Cloudbees has help but its very confusing. https://support.cloudbees.com/hc/en-us/articles/209715698-How-to-add-Java-arguments-to-Jenkins ) Mine was in /etc/sysconfig/jenkins (or jenkins-oc)

-Duser.timezone=America/Chicago

Note that this one has to be set when Java is first starting the JVM. As the Jenkins application come up, it reads the current time zone and stores it. If we were to use the Groovy method (as above) to set this Java system property, Jenkins would have already set the default time zone for all scheduled jobs and they would all use the default time zone NOT the time zone set in a Groovy init script.

3rd Change (optional)

Though I'm not sure if it matters, I also added this to Configure > Launch Method > JVM Options on my SSH slave. It depends on which kind of slaves you have if you want to find where to put that. And I don't know how to do it with a non-SSH slave or a Windows slave.

    -Duser.timezone=America/Chicago

Finally

Restart all the Jenkins and that's it.

More Stuff

The above will get jobs building on the correct hourly schedule(s). But perhaps you have one job that needs to run on a different schedule. Perhaps a long test run that needs to occur in the middle of the Indian night when the team in India is home sleeping.

You can put the following in for the schedule using two lines:

TZ=Asia/Kolkata
H H(0-1) * * *

Which will run the build job sometime between midnight an 1 AM using India's standard time zone.

Be careful that this long running build does not interfere with the work of people in other time zones. Midnight in India is 1:30 PM here (during the US Winder).