Multiple GIT tools in Jenkins Scripted Pipeline

In a Jenkins scripted pipeline script, perhaps a multi-branch pipeline job or a regular pipeline job, perhaps loaded from a Jenkinsfile in your repo or an inline script defined in the job config, this command will load a random git repo contents into your workspace

   git branch: 'master', credentialsId: 'GitHub-user-read', url: 'https://github.com/your-organization/your-repo.git'

When it runs, it will run using the 'git' tool that you set up under Manage Jenkins > Global Tools > Git (or on an Operations Center in a Global Tools menu element).

If you have more than one git tool, it will use the 1st one.

If you have more than one and want to use one that isn't first, you can't with the 'git' DSL step. But there is hope. I'll get to it below.

So imagine some situations:

• You have multiple versions of git installed on your node agents (or on the master if you run the jobs there). Each one is in its own path. For example, two version loaded in folders /opt/tools/git-1.9 and /opt/tools/git-2.12
• You have the same version of git installed on Linux and Windows node agents. For example, Linux is installed in /opt/tools/git/bin/git and Windows has it in c:\Program Files\Git\bin\git.exe as the two executable files.
• Similar to the last example, the executables are in the same places but the master is Linux and the node agent is Windows (or vice versa).

When you create any sort of non-pipeline job, you get to specify which version of git you need to load the code. You can pick the version that applies to the machine on which the executors reside -- Linux or Windows, Node Agent or Master. Or you pick the version of git that you need.

When you have a pipeline job using a Jenkinsfile (or other groovy script file), you again get the option to specify the git tool that is appropriate to load the repo from git. In this case, the Jenkinsfile is loaded on the master so you use the git tool with the right path to get to the master's copy of the git executable.

But in a script file, the 'git' DSL step works with the 1st defined git tool.

If you need to use the 2nd tool, change this:

    git branch: 'master', credentialsId: 'GitHub-user-read', url: 'https://github.com/your-organization/your-repo.git'

to this, adding the 'git-tool-name' as you specified it on the Global Tools configuration page:

    checkout([

        $class: 'GitSCM', 

        branches: [[name: '*/master']], 

        gitTool: 'git-tool-name', 

        userRemoteConfigs: [

            [

                credentialsId: 'GitHub-user-read', 

                url: 'https://github.com/your-organization/your-repo.git'

            ]

        ]

    ])

Jenkins Auto-Configuration of GitHub WebHooks to Initiate Build Jobs

Wow. What a mouthful of a title.

It turns out that Jenkins has the capability to make calls to GitHub and set up WebHooks. A GitHub WebHook is a thing that can be set up to issue an HTTP(S) call back to Jenkins whenever things happen to a particular GitHub repo. In particular, things like a push to GitHub or a merge of a Pull Request will cause GitHub to notify Jenkins. What we want is for Jenkins to then pull the new code and do a build, running tests, and all that.

I am using GitHub Enterprise. That means is is an independent GitHub server (not the one on the internet at github.com) and it is running in my employer's internal network. This GitHub instance is only available to us when at work or when connected to work via VPN.

In Jenkins, I am using Cloudbees Enterprise Jenkins (the version with an Operations Center and one or more controlled Client Master instances). But this should work with the open source version of Jenkins where there is only a Master. The reason its the same is that in Cloudbees Enterprise Jenkins the Masters work with GitHub and the Operations Center does not. It means you have to configure all your masters separately.

Configure to Auto-Manage GitHub Hooks

For each master, go to the main dashboard page.

• Manage Jenkins > Configure System > GitHub > Add GitHub Server > GitHub Server (Note: I have two places to enter a GitHub server on this page. We are using the one that
• GitHub > GitHub Server > API URL:  something like https://ghub.mycompany.com/api/v3
• Click Advanced (The one after the “Add GitHub Server” button)
• Additional Actions > Manage additional GitHub actions > Convert login and password to token
• GitHub API URL (again) : https://aaghub.qcorpaa.aa.com/api/v3
• Select radio button to enter it from login and password
• Manage Hooks: Click to select
• Enter a GitHub account id and password. The account should have permission to Admin any repos that you will set up to be automatically built when the content changes. (Perhaps less permissions than that are needed. Dunno.)
• Click Create token credentials to generate
• Make note of the generated credential’s name (it’s a sequence of numbers)
• Save
• Manage Jenkins > Configure System > GitHub Servers: Comes back to same screen
• GitHub Server > Credentials: Select the newly created credential (with the long name)
• Save

Now any jobs that are set to build on a change from GitHub (and that have a GitHub repo URL available on the config page) will have Jenkins configure GitHub to notify whenever that repo changes. (Note: This will not apply for GitHub Organization job types or Multi-Branch Build job types as they extend from a different subclass that does not have that code in it yet.)

Now we need to figure out if its working:

Go to the main dashboard page on the master OR click down from there to a folder where you can add a build job.

• New Item
• Name: TestJob1
• Click Freestyle Job
• Click OK

It will go the page where you add a freestyle job.

• Source code management > select GitHub
• Repository URL: enter one
• Credentials: Select the right one (I'm assuming you know how to access a GitHub repo code)
• Build Triggers > Select Build when a changed to pushed to GitHub
• For now select Build > Execute Shell (or Windows Batch Command if appropriate)
• In the shell script box, I put "ls -al" (or "dir", respectively).
• Save

Now to the the GetHub Web UI, log in using an account that has admin permissions on the repo you are using.

• Move through the web pages to the main page for the repo
• Click "Settings" at the right side of the top line of options
• Click "Hooks and Services" in the left, vertical menu
• Verify that there is a webhook pointing to your Jenkins server. It will use the URL that you configures as the URL of the server in "manage jenkins > configure system".

If so, you are golden. If not, you have a bunch of debugging to do. I don't know all the ways it can go wrong.

Maybe the Jenkins server cannot connect to the GitHub server over the internet. Go to a command line on the Jenkins server and ping or telnet to the GitHub server and see if it connects. 

Maybe the GitHub server cannot connect to the Jenkins server. Go to a command line on the GitHub server and see if you can ping or telnet to the Jenkins server. This may not be permitted. 

You can also test this by entering the web hook by hand in the GitHub repo and testing it. Note that if you leave the one you entered by hand, you will never know if the automatic method is going to work the next time you create a build job needing to auto-build from GitHub.

Maybe you go to the Manage Jenkins page and a red box is at the top telling you that some GitHub WebHooks didn't get created. Click to view this and you might get a clue as to what is wrong. If the credentials you selected when configuring the system under Manage Jenkins are invalid or do not have admin permissions in GitHub, this sort of error can happen.

That page you just viewed may have information about how to see the log messages relating to GitHub web hooks. Set up a logging context with these three items:

1. com.cloudbees.jenkins.GitHubPushTrigger
2. com.cloudbees.jenkins.GitHubWebHook
3. org.jenkinsci.plugins.github

Set the log level to "ALL" for these three.

Then go configure the test job again and save it again. This seems to retry setting of hooks sometimes. (But if you have messages about the hooks not being created, you will need to "ignore" each and then "disignore" each again.) I'm not really sure what it is that makes Jenkins try to create the webhook again if it doesn't work the first time. Sometimes I would just go build the test job and see if the hook got set up. Someone should document this.

Cloudbees Enterprise Jenkins Timezone When Machine is UTC

I did this on a Linux Red Hat Enterprise 7 machine. The problem I was having had to do with the default time zone on Linux was UTC. When I would do "time" from a command line, it would show UTC time. In my case, it was 6 hours later than the time on my phone.

Jenkins uses time in several ways and you have to make more than one change to make all of them use the correct time zone.

Note that there doesn't seem to be a way to have Jenkins figure out where the user sitting in front of a browser viewing a Jenkins screen may be located. If the server is in one time zone and the user is in another time zone, I'm hoping to have the time displayed equal the server's time zone.

Here are the places I found to check whether Jenkins is doing the time zone the way I want:

1. Go into a build job and look in the lower left margin. The time of the builds is shown there. It should be the time zone you want.
2. Go to a pipeline build job and look at the time displayed beside the flow diagrams for the builds. Again, this should be your desired time zone.
3. Go to the CJOC > Manage Jenkins > System Logs > All Jenkins Logs and notice the time of the last logged event. It should be in the desired time zone. If your server has been idle for a while, it might not contain a recent log message. Take this into account.
4. Repeat the same thing for the Client Masters. Each of them should display correct times in the log messages.
5. Go to any build job that runs at a particular time of day. On the Configure screen, there is an item for build periodically > schedule that shows one or more "cron" type strings. The 2nd column is for the hour. The message below the schedule box should show the correct time zones for the times the job was and will be run.

• A cron string "0 1 * * *" will run at 1 am every day. Check that's what it says below the schedule box.
• A cron string "H H(9-17) * * *" will run every day once per hour between 9 AM and 6 PM. The H means Jenkins gets to pick the minute value randomly.
• A cron string "@midnight" will run every night between midnight and 6 am.

1. Similarly, the times the jobs run (after they run and the time of running is in the left margin at the bottom on the build job page) should be correct in the desired time zone. You may also be able to check the time of build by looking at the dates on built artifacts--stuff like build log files and jar files and such.

1st Change

Create a file, $JENKINS_HOME/init.groovy.d/my-timezone.groovy (which Jenkins will run upon startup). Do this on the CJOC (Operations Center) machine and on any Client Masters.

System.setProperty('org.apache.commons.jelly.tags.fmt.timeZone', 'America/Chicago')

Note that you will use one of the Java timezone strings. I'm in the Central time zone of the United States. (See http://stackoverflow.com/questions/1694885/timezones-in-java)

This makes it so that Jenkins will use the defined time zone when it is formatting certain strings to contain the time of day of some event (like a log message or a build time). It sort of depends on which bit of code is doing the formatting.

2nd Change

Then you need to add a startup string to the sysconfig file for jenkins. (How do you find it? Cloudbees has help but its very confusing. https://support.cloudbees.com/hc/en-us/articles/209715698-How-to-add-Java-arguments-to-Jenkins ) Mine was in /etc/sysconfig/jenkins (or jenkins-oc)

-Duser.timezone=America/Chicago

Note that this one has to be set when Java is first starting the JVM. As the Jenkins application come up, it reads the current time zone and stores it. If we were to use the Groovy method (as above) to set this Java system property, Jenkins would have already set the default time zone for all scheduled jobs and they would all use the default time zone NOT the time zone set in a Groovy init script.

3rd Change (optional)

Though I'm not sure if it matters, I also added this to Configure > Launch Method > JVM Options on my SSH slave. It depends on which kind of slaves you have if you want to find where to put that. And I don't know how to do it with a non-SSH slave or a Windows slave.

    -Duser.timezone=America/Chicago

Finally

Restart all the Jenkins and that's it.

More Stuff

The above will get jobs building on the correct hourly schedule(s). But perhaps you have one job that needs to run on a different schedule. Perhaps a long test run that needs to occur in the middle of the Indian night when the team in India is home sleeping.

You can put the following in for the schedule using two lines:

TZ=Asia/Kolkata
H H(0-1) * * *

Which will run the build job sometime between midnight an 1 AM using India's standard time zone.

Be careful that this long running build does not interfere with the work of people in other time zones. Midnight in India is 1:30 PM here (during the US Winder).

Jenkins Pipeline Shared Groovy Script by Hand

There is currently a new version of the Pipeline Shared Groovy Libraries Plugin (2.5) that adds capabilities to put some Groovy scripts in Git, at least, and to make them available easily to call from Jenkinsfile (the Jenkins Pipeline script, also stored with the application code in Git)

I can't move to that version yet and so I set it up to load the script by hand:

def notify

stage "Load-Scripts"
node() {
    git branch: 'master', changelog: false, credentialsId: 'ID', poll: false, url: 'https://github.company.com/my-organization/my-repository.git'
    notify = load 'src/com/aa/jenkins/pipeline/notify.groovy'

    notify.notifySlack("START")
}

The above code goes in the Jenkinsfile at the top, before any other  stages. You will, of course have to put in your own  credentials and branch and so forth, the Pipeline Snippets tool might be handy in doing that line.

Some things ...

The variable into which we store the class created by loading the one groovy file has to be defined with the 'def' and without a type. This makes it into a global variable that can be used outside the scope in which we do the load.

The path to the groovy file is in the context of the root of the git workspace.

You may need a node name inside the parentheses.

New to Git. Used to SVN.

Some random thoughts about moving to "git" from "SVN":

In git, when they talk about branches and draw those little diagrams with circles and lines to show branching, it is incredibly complex because git encourages branching.

When you do a commit on your local copy of the repo, it creates a new thing that is called a branch according to the way git thinks.

They draw pictures like this:

Each circle is what happens when there is a git commit and is called a branch. 

In SVN its just a sequence of commits and they tend to be called versions or revisions. In SVN they have numbers (in sequence) but the branches are named with their own sequence of numbers.

In git, they also talk about the contents of a branch which seems to mean all the changes that comprise the branch.

Context in Javascript regex replace with anonymous function

I know. The title is a killer. I'm only putting this post here since I couldn't find this information anywhere.

---

First, lets talk about the 'replace' function on Javascript strings. Here's an example with a search for a string.

var name= "World";
var msg = "Hello name!";
var result = msg.replace("name", name);

This results in:

Hello World!

---

Now, you can use a regular expression (and, in fact, probably should) in place of the string. The yellow highlight shows what you change to make it a regular expression. (I'm not going to define regular expressions. I'm trying to get to my real concern in this particularly wordy way.)

var name= "World";
var msg = "Hello name!";
var result = msg.replace(/name/, name);

This results in the same thing:

Hello World!

---

Of course, we can use the regular expression to search for something more complicated. In this case, it will find one or more vowels between the 'n' and the 'm' in 'name' and match the whole word.

var name= "World";
var msg = "Hello name! Hello nime! Hello nouiame!";
var result = msg.replace(/n[aeiou]+me/, name);

This results in the same thing three times even though the starting strings contained different vowels. Oh, no, it doesn't. It only replaces the first occurrence:

Hello World! Hello nime! Hello nouiame!

---

We get the result we want if we add the 'g' option to the regular expression.

var name= "World";
var msg = "Hello name! Hello nime! Hello nouiame!";
var result = msg.replace(/n[aeiou]+me/g, name);

This results in the same thing three times even though the starting strings contained different vowels and it really works:

Hello World! >Hello World! Hello World!

---

Finally, getting close to what this blog entry is about. It concerns you when you have a block of Javascript like this. You are replacing multiple instances of some matched regular expression using a function to generate the replacement string. In this case we will reverse the original string and include that in the replacement.

var msg = "Hello name! Hello nime! Hello nouiame!";
var result = msg.replace(/n[aeiou]+me/g, function(matched) {
    return "World (" + matched.split("").reverse().join("") + ")";
});

This results in the same thing three times even though the starting strings contained different vowels. Oh, no, it doesn't. It only replaces the first occurrence:

Hello World (eman)! Hello World (emin)! Hello World (emaiuon)!

---

Finally, what I want to comment on. Again we will reverse the original string and include that in the replacement. But lets put it in a function and get the "World" from a variable that is in the same context.

{
    worldString : "World",
    myFunction: function() {
        var msg = "Hello name! Hello nime! Hello nouiame!";
        var result = msg.replace(/n[aeiou]+me/g, function(matched) {
            return this.worldString + " (" + matched.split("").reverse().join("") + ")";
        });
        return result;
    }
}

This should result in the same thing (You know, three times.) But it actually fails because the context of the function called by "replace" is the main 'window' object. Here is what Chrome (Version 23.0.1271.64 m), IE 9 and FireFox (16.0.2) shows:

Hello undefined (eman)! Hello undefined (emin)! Hello undefined (emaiuon)!

Here is a jsfiddle showing this fail and the success

---

So, what works? We do put it in a function still and we get the "World" from a variable that is in the same context. But we have to save the context in "myFunction" and use it in the anonymous function:

{
    worldString : "World",
    myFunction: function() {

        var self = this;
        var msg = "Hello name! Hello nime! Hello nouiame!";
        var result = msg.replace(/n[aeiou]+me/g, function(matched) {
            return self.worldString + " (" + matched.split("").reverse().join("") + ")";
        });
        return result;
    }
}

This does result in the three times we expected last time:

Hello World (eman)! Hello World (emin)! Hello World (emaiuon)!

Here is the same jsfiddle showing the fail and this success

Maven Checkstyle Reports Shortcut

I'm using Maven 2.2.1 today. MyEclipse version is 8.5.

When you want a Checkstyle report from Maven 2 you need to put these two bits of xml in your pom.xml. Then you need to create the Checkstyle.xml and license.txt files.

Be sure to check what the right version is of the plugin. Its shown at the bottom of the page: maven-checkstyle-plugin You might have to back off one version.


<build>
    <pluginManagement>
        <plugins>
            ...
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-checkstyle-plugin</artifactId>
                <version>2.6</version>
            </plugin>
            ...
         </plugins>
    </pluginManagement>
</build>


and then put the report request in. (Be sure to put it after the JXR report if you are using that one. There is some report that needs to be after JXR and it might be Checkstyle. I can't remember for sure but we all have our little superstitions.)


<reporting>
    <plugins>
        ...
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-checkstyle-plugin</artifactId>
            <configuration>
                <enableRulesSummary>false</enableRulesSummary>
                <configLocation>${basedir}/Checkstyle.xml</configLocation>
                <headerLocation>${basedir}/license.txt</headerLocation>
                <excludes>**/generated-sources/**/*.java</excludes>
            </configuration>
        </plugin>
        ...
     </plugins>
</reporting>


You may (or may not) want the "enableRulesSummary" option off. I find it obnoxious to have all that page space taken up by the long explanations.

The "license.txt" file is the easiest. You only need it if you are going to have a rule that checks for a certain header at the top of each .java file. You can change the name but be sure to use the real name when setting up the Checkstyle rules that you want to use.

Now, for the rules file -- Checkstyle.xml. There is an easy way to create this file. Much easier than typing a bunch of cryptic XML into an editor. We just let the Eclipse do it.

You have to have the Eclipse Checkstyle plugin installed. I've done it multiple times with multiple versions of Eclipse and multiple versions of the Checkstyle plugin. Once you have that plugin installed, use the Eclipse preferences menu item to bring up the preferences box. Find the Checkstyle entry in the left section of the box and click it.

This will bring up the Checkstyle preference. Create a Checkstyle configuration of your own. Or you can use the built-in Sun configuration as a start and fool around until it tells you enough but not too much. That's always the problem with code analysis tools. When you first run them on some code, they complain about all sorts of things that you just don't care about.

Maybe your code has curly-braces in the "wrong" place and you don't really care. Change the rule settings to ignore placement of the curly-braces. So, click about 1000 times and get all the Checkstyle rules set up as you like.

On a side note: I have found that using the Checkstyle Eclipse plugin together with the Eclipse Code Formatter you can sort of play the tools off against each other for code format issues.

Most groups of developers have some sort of code-format conventions. Whatever it is, most everyone dislikes some aspect of it. But if you supply them all with a code formatting template created to be used in Eclipse and exported to a file, they can use it just before submitting code.

They can type however they want. They can even format it to their own preferences while working on it (for the fussy programmer). Just fix it before putting the code away. Then use Checkstyle to see if they are doing it. You can make a build fail if the format is wrong but I find that obnoxious. Just create a report somewhere that the managers or team leads can look at and complain now and again. In real life, most everyone just gets used to it unless they are working on multiple sets of code with different rules.

Anyway, put the formatting rules in your Eclipse and format a large body of code. Do that by right clicking on a source folder and finding "reformat" on one of the nested menus. Then let Checkstyle, also inside Eclipse, check it.and see what the errors are. You will probably end up changing both configurations until you get it right.

Whatever the rules, when you have them "right," you can just export the rules with the button on the Checkstyle preferences page. Store them, if you want to use my pom XML as shown above, in the same folder that the pom.xml file is in and name that export file "Checkstyle.xml" matching the letter case. (Windows doesn't care but you'll thank me for that suggestion if you ever build under Linux.)

The trick is that the "export" stores the file in exactly the format that Checkstyle's Maven 2 plugin wants (or even the same format that Checkstyle's ANT task wants). I keep Checkstyle.xml with the project code, store it in my version control and now and then make a few changes.

The thing is, as time goes by you either get tired of seeing some of Checkstyles complaints that you thought you cared about. Or you decided to ratchet it up a notch and add some more checks because the code is better than it used to be or because the team is getting into some bad habits that you'd like to nip in the bud.

For example, I worked with a team once and a couple of guys couldn't find it in themselves to put a debug logging statement in the code they were working on. They had to put a "System.out.println()" call in there and, of course, they would forget to remove them all. We added a custom regex rule to Checkstyle that said "System.out" and "printStackTrace(" were illegal to use in the code. We had to live with a few complaints here and there where we really wanted to output to the console. One was a little utility that just lived in the code but ran from the command line. The other was where everything went horribly wrong during startup and we needed a message but the logger wasn't set up yet. So we found all the funky debug lines and had the offenders (who we identified from the checkin records) remove them or change them to use the logger.

Once you save the export file, you run "mvn clean site" to generate the report.

If you have a site deployment location set up in the pom.xml you can then do "mvn site:deploy". Note that I have found you shouldn't do these three targets all together. Do the clean-site first and then do the site:deploy. It just works better that way.