Using Gson in your e2e tests might result in tests passing even when contract is broken. Gson is quite imprecise when it comes to deserializing objects from json. Is there any difference between an enum and a string? Is there any difference between s string and a number? Using Java you might answer: sure there is. Even when talking about JSON you’ll answer: yeah there is a difference. But for GSON? Meh whatever.

Gson will do its best not fail when it can not deserialize object it’ll do the best it can not to fail. For example when it can not resolve enum value for string it’ll be sneaky about it and will ignore it without word of warning. There will be no error. No warning. Nothing. Everything is fine. Because of the feature|bug implemented in Gson it might not be good idea to use it for contract testing.

I’ve been migrating e2e tests from Gson to Jackson because of @JsonProperty definitions generated by OpenApi JAXRS generator. After I’ve finished configuring it there were still two tests failing. I’ve fixed all compilation errors, and I haven’t changed API implementation at all but one test was failing. It should just work because all I did was to change serialization library and nothing more. It was enough for all other tests…​

After some debugging and reading returned JSON I’ve found the issue. It turned out there was a custom @ExceptionHandler added later which returned an object similar to RegularError API error message but slightly different let’s call it CustomError. RegularError object has errorCode which is one of the strictly defined enum values. CustomError also has errorCode but it’s a simple string in this case names of the constraints that failed validation. All other fields are the same.

We were using Gson for object serialization and deserialization in end-to-end tests. Idea was to use a different serialization library in tests to be sure everything works as expected and nothing is lost (or unexpectedly changed) when pushing objects through cable. It turned out Gson doesn’t work in a way I expected it to work.

Let’s define two objects:

and

Very similar but not quite the same. For those objects let’s use following json representation:

SampleObjectJson:

{
    "message":"Hello World",
    "code": "OK",
    "extraProperty": "cool"
}

OtherObjectJson:

{
    "message": "Other Hello World",
    "code": "All Good",
    "extraProperty": 42
}

Let’s see how Jackson’s ObjectMapper will react to misplacing them:

@Test
public void jackson_fails_when_deserializing_wrong_object() {
    assertThrows(
            InvalidFormatException.class,
            () -> objectMapper.readValue(OtherObjectJson, SampleObject.class)
    );
}

Which will fail with (if you let it):

This is something I expect from serialization library. When it can not deserialize json just throw exception and let someone else handle it.

Now Gson turn.

@Test
public void gson_works_when_deserializing_wrong_object() {
    var sampleObject = gson.fromJson(OtherObjectJson, SampleObject.class);
    assertNull(sampleObject.code);
    assertEquals("Other Hello World", sampleObject.message);
    assertEquals("42", sampleObject.extraProperty);
    assertNull(sampleObject.code);
}

Not exactly what I expected.

In some cases, it might be what you need but that’s not what I want in end-to-end tests. I made some effort with those tests - I wrote them. Then I decided to use statically typed language. Once I need for them to payout on invested time and catch the bug Gson says: "Meh, whatever it’s kinda working. Don’t worry about it". Exactly the opposite of what I want it to do. I was expecting big red notification on CI server and an email with information that CI pipeline is broken. I got nothing but a false positive test.

I’m complaining about Gson here but truth be told it’s a human error. You can treat this as a bug - https://github.com/google/gson/issues/188, but now after 7 years this is more like a feature…​

Lesson is simple. Read the documentation of tools you use or test if they behave in a way you expect them to behave and do not take things for granted. 15 minutes saved at the beginning of the project result with WorkDetailsError when someone tries to signup to the application :)