01 December 2009

Hudson updates, Linux and the corporate proxy

My Hudson build server is running on Red Hat Linux. This became important when I figured out that I couldn't update Hudson automatically because of some sort of proxy problem.

I would select Manage Hudson > Manage Plugins and then select a plugin or two to update and click the "Install" button to do it. It would give me an error "NoRouteToHost" and show a Java stack trace.

First Try

I was running Hudson 1.309 and, though it might be a bug in that version (but don't quote me on that), entering the corporate proxy server's IP and port on the "advanced" tab of that "Manage Plugins" page didn't fix the problem. (There was a wierd bug where, if I used FireFox 3 to enter the proxy info, it would default the user and password to those I used to log into Hudson. This didn't happen on IE 6.)

This seemed to make no difference at all. The same error popped up as before.

Second Try

I use the command line to start Hudson. Its a command like "java -jar ..." with a zillion options included. So I added "-Dhttp.proxyHost= and -Dhttp.proxyPort=" to the command line.

This had a different result. The plugin upload would work but Hudson could no longer access my Subversion server. The error message appeared in the console log when a job would run but it still said "No route to host".

Third Try

Well, I thought, I will need to exclude the Subversion server from Java's proxy information. This should have worked by adding "-Dhttp.nonProxyHosts=*.domain.com" to the command line.

This didn't work either. I don't know why not but the errors were the same as in the 2nd try.

I took all that stuff off the command line.

Fourth Try

Now, I though, isn't there some way to tell Linux about a proxy to use by adding an environment variable of some sort. I had to Google around to find it. You add an entry to the environment with a command like "export HTTP_PROXY=http://proxy.domain.com:8080/" if you are just typing. I left off the "export" part for the shell script that starts Hudson running.

This worked. Things would update inside Hudson. The jobs can get to the Subversion server. All seems to be cool now.

Disclaimer

I don't claim to know why the last method worked. I don't know if it would work the same way if I was using Tomcat to run the hudson.war. I don't know if the "advanced" tab's proxy settings work better in a later release of Hudson.

All this ignorance on my part bothers me. But, for now, I'm just happy it works.

-- Lee

20 August 2009

JMock vs. Mockito


I heard about Mockito, a new-to-me framework for writing unit tests with Mock objects, and decided to take it for a spin. Below I have shown the same test written with JMock (version 1) and Mockito (version 1.8).

Now, I know that JMock version 1 is old but the real tests for the real code have to run in Java 1.4 so that's the version we tend to use.

Here is the situation:

* I have a list of "alerts"
* Each alert is a simple form of a canned message. The object contains an email address, a message name and some parameters that are used to fill in the blank spots in the message.
* Each parameter is just a name-value pair.
* The message may have some of the names encode in it's text and, for each recipient, his own parameter values are filled in.
* I have to validate the alerts in several ways. The class to do one of those validations is the class under test.
* The validation process is to look through the list of alerts and throw out any that aren't valid. The reason they are invalid is coded into the particular validator class's code.
* Validations can be chained so this validator can have another validator injected that will be called to do further validations when this one is done.
* The validator that is being tested checks whether the exact same alert was previously sent to the same person. (It compares the parameters, message name and recipient to see about that.) It does this by looking for a matching alert in a database containing recently sent alerts.
* If it finds that the alert is a duplicate of one recently sent, it updates the date in the database to indicate an attempt was made today to send it again.
* Some other code fills the database after the alerts get sent successfully.
* Some other, other code cleans out old records from the database to give definition to the term "recent."

So, we are testing the AlertDuplicateRemover so here is one test to see what happens when there is one alert in the list and it is a duplicate.

JMock first:


public final void testStoreSentAlertsDuplicated()
{
int alertCount = 1;
AlertResultBO alertFeed = AlertBOMother.getAlertWithValidParms(alertCount);

Mock duplicatesDaoMock = mock(AlertDuplicatesDao.class);
for (int i = 0; i < alertCount; ++i)
{
duplicatesDaoMock.expects(once()).method("get").with(eq(alertFeed.getAlerts()[i])).will(
returnValue(alertFeed.getAlerts()[i]));
duplicatesDaoMock.expects(once()).method("update").with(eq(alertFeed.getAlerts()[i]),
isA(Calendar.class));
}
AlertDuplicatesDao duplicatesDao = (AlertDuplicatesDao)duplicatesDaoMock.proxy();

Mock validatorMock = mock(AlertValidator.class);
validatorMock.expects(once()).method("validate").with(isA(AlertResultBO.class)).will(
returnValue(alertFeed));
AlertValidator validator = (AlertValidator)validatorMock.proxy();

AlertDuplicateRemoverImpl remover = new AlertDuplicateRemoverImpl();
remover.setAlertValidator(validator);
remover.setAlertDuplicatesDao(duplicatesDao);

remover.validate(alertFeed, CalendarMother.yesterday());
}


And then Mockito


public final void testStoreSentAlertsDuplicated()
{
int alertCount = 1;
AlertResultBO alertFeed = AlertBOMother.getAlertWithValidParms(alertCount);

AlertDuplicatesDao duplicatesDao = mock(AlertDuplicatesDao.class);
when(duplicatesDao.get(alertFeed.getAlerts()[0])).thenReturn(alertFeed.getAlerts()[0]);

AlertValidator validator = mock(AlertValidator.class);
when(validator.validate(any(AlertResultBO.class))).thenReturn(alertFeed);

AlertDuplicateRemoverImpl remover = new AlertDuplicateRemoverImpl();
remover.setAlertValidator(validator);
remover.setAlertDuplicatesDao(duplicatesDao);

remover.validate(alertFeed, CalendarMother.yesterday());

verify(validator, times(1)).validate((AlertResultBO)any(AlertResultBO.class));
verify(duplicatesDao, times(alertCount)).get((AlertBO)any(AlertBO.class));
verify(duplicatesDao, times(alertCount)).update(any(AlertBO.class), any(Calendar.class));
}


Notes on the code:

AlertBOMother - I got the idea for this from Derek Lane a few years back. Its a class used only for testing. It has static methods that return prepopulated objects so that multiple tests can just ask for one. Its particularly useful to clean up all that code where you fill in an object's properties. Using the "Mother" hides that away so it doesn't clutter the test (and hide what its testing). It also makes the prefilled object available for multiple tests without using RBCAP (reuse by cut and paste). When properties for the object change or are added, using a 'mother' makes it easy to adjust the tests. Finally, it makes it easy to start your testing with a simple version of the object and then improve the contents used for testing later.

Both mock the DAO that is used to read (get) and update the database records. 'get' is used to see if the alert is already there (implying it was already sent). A null is returned if no match is found in the database.

Both also mock the other validator to which this validator chains after working its magic.

The mocks, in both cases, are injected into the class-under-test as they would by by Spring in real life. That class knows nothing about Spring so there is no need to call any of Springs special methods to do Spring stuff.

Mockito code seems a little simpler. There is a bit less clutter to hide the intent of the test.

I never did like how JMock has you put a method name in a string. Mockito doesn't do this.

JMock sets up the expectations ahead of time. Mockito checks the expectations after the fact. That makes Mockito more like the way JUnit works. In both, you do stuff and then see if the results are right.

JMock sort of mixes up the expectation code and the setup code. Let me explain. There is code to tell the mock object what to do. That's what I call 'setup code'. Three is code to check that the expectations of what happened during the running of the code really happened. That's the 'expectation code.' Notice that in JMock, both types of code preceed the test and are mixed together. In Mockito the setup preceeds the test and the expectations follow it.

Interestingly enough, running inside Eclipse 3.3 under Java 6, the Mockito tests took twice as long to run as the JMock tests. This is the time for all the tests not just the one test shown above. The difference was only 1/2 second in this case but it could add up. I'm not sure how much it matters as I find that only about 10-15% of my tests need mocks anyway. (But, maybe I need to write more tests.)

Neither framework has very good documentation. In both cases, it consists of a few pages of examples and some javadoc. (I should note that I'm talking about JMock 1 here. The docs for JMock 2 may be better.) JMock is here http://www.jmock.org/index.html and Mockito is here http://mockito.org/

Finally, I don't pretend to be an expert in either framework. I've been using JMock on and off for 4 years on several projects. This is my first attempt with Mockito.

I'm sure there are better ways to use both and one or the other may shine when the tests are more complicated. (On the other hand, complicated tests are a pain in the ... neck and I really like simple ones better.)

01 July 2009

Interesting Web Service Bug

The other day we found an interesting bug in using a SOAP web service. Of course, it was only found after totally fouling things up for two days leaving two more days of things to clean up for the whole team.

I'll tell you up front what was wrong (but skip to the next paragraph is you want a mystery). The two ends of a web service (client and server) had different ideas about the allowable range of values that were allowable for an integral value passed back and forth.

The key thing you need to know is that our application makes a call to an operation on a web service at another company. We defined the service and had them implement to our WSDL. A list of data elements builds up in their system until we ask for it. The web service provides us with a block of those element that we then process.

When we request, data comes in a block along with a 'handle.' The handle is defined as an xsi:int element in the schema. We process the data. It takes about 5 minutes, usually. Then we call another operation on their web service, passing back the handle, to tell them that the block they gave us with that handle is done. They can remove it from the list of elements that are waiting to come to us.

Then we loop around and request the next block of data and do the same thing process again. When there is no more data to get, they return an empty block. When we see the empty block, we know there's no more data for today. So, we wait until tomorrow and start over.

The bug was rooted in a misunderstanding about what allowable values were for the handle. When they sent us a block, they got a unique number out of their database. I don't know what the range of integral values that data element allowed but it was larger that the Java 'int' that we were using to hold the handle. Further, I haven't looked up to see if an xsi:int in a schema in a SOAP WSDL has a defined range anyway. So I don't know who had the range wrong, if anyone. Truly, I don't care, but don't tell anyone.

So, everything ran fine for years. Literally!

Then, one day, they sent us a number larger than the largest allowable positive Java 'int'. Their datatype had been converted to a string in the XML of the SOAP message. It was just a sequence of digits. Our web service framework (Axis 1.3) converted it to Java 'int' and came up with a negative value. (The framework code converting it did one last mathematical operation that overflowed the signed 32-bit number.)

When we sent the value back to them, they didn't recognize it as any existing handle so they ignored it. (I suppose they might have logged an error or even started the red lights flashing all over the developer's cubes. I don't know.) The block of elements they sent us were not removed from the list. When we asked for the next block and got the same elements again.

This repeated processing of the same elements happened repeatedly and repeatedly and repeatedly for 17 hours at 10 minute intervals before we got it stopped. (Things like this never happen until you just left the office for the day.) It left us lots to clean up. (I'm just glad it didn't start happening on Friday evening.)

You might wonder about the results. Well, some people got confused for a few hours when they saw duplicate data. We got everything fixed to not show any bad stuff to the users quickly. Pretty soon we got our integer ranges aligned with the other company and the problem quit happening. A few days later it was just a bad memory.

Anyway, it seemed interesting to me that a simple thing like a misunderstanding of the data range of integral values in a web service hid away for years and then rose up to bite us in such a big way.

It just underscores the need to define more than the interface for a web service. You have to define what the data means and what are allowable values if you want to cover all the bases, so to speak.

It also underscores the difficulty in testing for this sort of inter-system bug. I'm not sure how one would create a "unit" test that would just test this sort of interface. There are two systems. Two sets of developers. Two companies. (Two countries, actually.) And little shared information on either system's internal workings.

19 June 2009

Hudson, Maven and Nexus Credentials

I decided to figure out how to post my Logger project from Hudson to Nexus.

Outline this time (I'm in a hurry)

* Go to Nexus which, in my case, is http://nexus.flatland.com:8090/nexus.
* Login as Admin
* Click "user" in the left sidebar
* Click add a user in the top of the main window
* I added one called 'hudson'.
* I gave it two permissions. 'Nexus Deployment Role' and 'Repo: All repositories (full control'
* It has to be active but the other fields are self-evident.
* Save the changes.

Now you have to make sure the repo can be deployed to:

* Still logged into Nexus as Admin.
* Click "Repositories" in the left sidebar.
* Choose 'releases' at the top (or 'snapshots' if you want to deploy a snapshot)
* Choose the 'configuration' tab
* Maybe you scroll down, maybe not, but get to the "Allow deployments" entry and make sure it is 'true'.
* If you changed it, you have to 'save' the result.

On the build server you will need to edit settings.xml to add credentials for Hudson to use with Nexus. On my Red Hat Linux server it is in ~/.m2/settings.xml (/home/hudson/.m2/settings.xml on my server).

This should go into the settings.xml file. It goes in the <servers> section.

<server>
<id>nexus.flatland.com</id>
<username>hudson</username>
<password>password</password>
</server>

Now Nexus will let the 'hudson' user deploy something.

* Go to Hudson.
* Select the Logging job.
* Click on one of the builds in the list of builds at the bottom of the left sidebar. (Usually you will want to pick the most recent build but older ones will let you restore something older.)
* Click "Redeploy Artifacts" among the left sidebar options. It has a green curvy arrow icon.
* Enter (or use the down arrow to get the list) the Nexus repo URL. It will look something like this: http://nexus.flatland.com:8090/nexus/content/repositories/releases (but use 'snapshots' instead of 'releases' if you are deploying a snapshot). The host name is the same as the Nexus server that I messed with above.
* Click 'advanced' on the right side of the screen.
* Enter 'nexus' for the id.
* Click 'ok' and it should deploy.

Note that all the 'artifacts' created with the Maven build will be deployed. This might include 'sources', 'test-jar' and 'test-sources'. Adding the 'test-jar' is done with an option on the maven jar plugin. The other two are don with options on the maven sources plugin.

Hudson, Maven and SVN Credentials

I figured out today how to better configure it so a Maven build running on Hudson would be able to access SVN with a user.
  • Hudson - Continuous integration build server http://hudson.dev.java.net

  • Maven - Build tool for Java applications (among others) http://maven.apache.org/

  • SVN - Subversion is a source code repository used to store source code (and what-not) and keep all the various versions created while the software is developed and maintained. http://subversion.tigris.org/ (I use the version from http://www.collab.net/ where you can get binaries for Windows, Linux MacOS and Solaris.)
I'm in the process of moving things from Continuum (another CI build server) to Hudson and still evaluating things.

This all started when I added a build job to Hudson for a library we use internally here where I work. This new Logger project holds some add-on things to use with Log4j.

When I started, the other day, putting jobs in, I entered the SVN repo URL for Hudson to use when it updates its workspace copy of the source code. When I moved the cursor off the URL box, Hudson complained that it couldn't use that repository and suggested that I enter user credentials. I entered my personal login and password and hurried on. I was trying to get something to build on this CI server I was testing.

The other builds worked. When I added the Logger job, it failed when trying to do Maven's Change Log report. Here is the error:
[INFO] Generating "Change Log" report.
[INFO] Generating changed sets xml to: /hudson/jobs/Logger/workspace/Logger/target/changelog.xml
[INFO] Executing: svn --non-interactive log -v -r "{2009-05-19 19:45:12 +0000}:{2009-06-19 19:45:12 +0000}" http://svn.flatland.com/repos/trunk/base/Logger/
[INFO] Working directory: /hudson/jobs/Logger/workspace/Logger
[ERROR] Provider message:
[ERROR] The svn command failed.
[ERROR] Command output:
[ERROR] svn: error: cannot set LC_ALL locale
svn: error: environment variable LANG is en_US.UTF-8
svn: error: please check that your locale name is correct
Now, after a whole bunch of googling I came to the conclusion that I needed to update the Subversion command line client version from 1.3.? to a newer version. The current version on Collabnet's site is 1.6.? so I downloaded it, uninstalled the old one and installed the new one. (The server is Red Hat Linux 4.? so this was all done with RPM commands and changing some environment variables and the path.)

Ok ... cool. Now that error is gone. I build and ...

[INFO] Generating "Change Log" report.
[INFO] Generating changed sets xml to: hudson/jobs/Logger/workspace/Logger/target/changelog.xml
[INFO] Executing: svn --non-interactive log -v -r "{2009-05-20 16:42:57 +0000}:{2009-06-20 16:42:57 +0000}" http://svn.flatland.com/repos/trunk/base/Logger/
[INFO] Working directory: /hudson/jobs/Logger/workspace/Logger
[ERROR] Provider message:
[ERROR] The svn command failed.
[ERROR] Command output:
[ERROR] svn: OPTIONS of 'http://svn.flatland.com/repos/trunk/base/Logger': authorization failed: Could not authenticate to server: rejected Basic challenge (http://svn.flatland.com)
Now, I'm not getting the right credentials to SVN.

It turns out that there are two places where credentials are stored in this case.
  1. Hudson has credentials that it uses when updating the source code prior to a build. It also uses it to check whether a build is needed if you check that box.

  2. Maven has credentials that is uses when it is accessing SVN.
I decided to just set up a new credential for Hudson to use with SVN. So I went over to the SVN server and added a new user, "hudson", and gave him permission to read and write to the interesting parts of the repo.

Then I had to go into Hudson, select the Logger job, select to configure, scroll down to the SVN repo URL and click the little help icon to the right of it. It is a blue circle with a tiny question mark in the middle. Inside the message there is link in the 2nd paragraph where it says, "If you already have a working credential but would like to change it for other reasons, click this link and specify different credential." Click the words "this link".

This brings up a new page where you can enter the first part of a URL for the SVN repo. I enter "http://svn.flatland.com/repos/trunk" but I could have added "Logger" on the end to only apply to this project's code. Then I selected the top radio button and entered the new user name (hudson) and password (think again). Clinking OK saved that away somewhere within the depths of Hudson. I don't know if there is a way to see all the ones that have been set and delete of update them.

Now, there is one more part of this. Or is it two parts that work together.

I have a part in the pom.xml for this project that tells where Subversion is. It's in the <scm> section in the pom and looks like this. I didn't have to change it but I did have to look at it to see what was in there. This tells Maven where the repo is. It gets used in several ways.
<scm>
<connection>scm:svn:http://svn.flatland.com/repos/trunk/base/Logger/</connection>
<developerconnection>
scm:svn:http://svn.flatland.com/repos/trunk/base/Logger
</developerconnection>
<url></url>
</scm>
And I have to put something in the settings.xml file on the build server. It is in ~/.m2/settings.xml for the user that Hudson runs under on the build server. In my case that is /home/hudson/.m2/settings.xml. Then I add a new bit of credential that looks like this:
<settings>
...
<servers>
... other server tags ...
<server>
<id>svn.flatland.com</id>
<username>hudson</username>
<password>password</password>
</server>
</servers>
...
</settings>
Save that and the buld now works.
[INFO] Generating "Change Log" report.
[INFO] Generating changed sets xml to: /hudson/jobs/Logger/workspace/Logger/target/changelog.xml
[INFO] Executing: svn --username hudson --password ***** --non-interactive log -v -r "{2009-05-20 17:27:23 +0000}:{2009-06-20 17:27:23 +0000}" http://svn.flatland.com/repos//trunk/base/Logger/
[INFO] Working directory: /hudson/jobs/Logger/workspace/Logger
[INFO] Generating "Developer Activity" report.
[INFO] Using existing changelog.xml...
[INFO] Generating "File Activity" report.
[INFO] Using existing changelog.xml...
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------

BTW, I'm using Hudson 1.309 version.

-- Lee Meador