Apache POI Excel Example

Learn how to create an Excel spreadsheet from within Java using Apache POI, a library for working with Microsoft Documents.

1. Introduction

Apache POI is a Java library to read and write Microsoft Documents including Word and Excel. Java Excel API can read and write Excel 97-2003 XLS files and also Excel 2007+ XLSX files. In this article, we show how to get going using the Apache POI library to work with Excel files.

Continue reading “Apache POI Excel Example”

Jackson Tree Model

1. Introduction

Jackson represents the JSON object model as a tree of JsonNode objects. This is called the Tree Model since it comprises of a tree of JsonNodes, including ObjectNode and ArrayNode. The tree mirrors the structure of the JSON document. It can be created by parsing a JSON document or by creating an in-memory representation.

2. Parsing a JSON File

A JSON file can be parsed and a tree model representation can be created using the ObjectMapper.readTree().

ObjectMapper mapper = new ObjectMapper();
JsonNode root = mapper.readTree(new File(args[0]));

The resulting tree model can be converted back to a JSON document as follows:

System.out.println(mapper.writeValueAsString(root));

3. Convert a Java Object (POJO) to JsonNode

Let us learn how to build a tree model from a POJO (Plain Old Java Object). Consider these classes:

public class Address {
    private String address1;
    private String address2;
    private String city;
    private String state;
    private String zip;
}

public class User
{
    private String firstName;
    private String lastName;
    private Address address;
}

Instantiate the objects as required:

User user = new User();
user.setFirstName("Harrison");
user.setLastName("Ford");
Address address = new Address();
address.setAddress1("123 Main Street");
address.setCity("Hollywood");
address.setState("CA");
address.setZip("33023");
user.setAddress(address);

A single function call suffices to convert the POJO to a JsonNode.

JsonNode root = mapper.valueToTree(user);
System.out.println(mapper.writeValueAsString(root));

The output JSON is:

{
  "firstName" : "Harrison",
  "lastName" : "Ford",
  "address" : {
    "address1" : "123 Main Street",
    "address2" : null,
    "city" : "Hollywood",
    "state" : "CA",
    "zip" : "33023"
  }
}

4. Convert a JsonNode to POJO

Once you have assembled a tree model from the JsonNode instances, how can you convert it to a POJO? Quite easy and all the contained JsonNode instances are also converted properly.

User user = mapper.treeToValue(root, User.class);

5. Convert Generic Containers to JsonNode

You can build up an object representation from nothing more than Java’s generic containers such as Map and List. Is it possible to convert such a representation to a tree model (JsonNode)? Of course it is. Use ObjectMapper.valueToTree() as before.

Map<String, Object> user = new HashMap<>();
user.put("firstName", "Harrison");
user.put("lastName", "Ford");
user.put("emailAddress", Arrays.asList("harrison@example.com",
              "hford@actors.com"));

Map<String, Object> addr = new HashMap<>();
addr.put("address1", "123 Main Street");
addr.put("address2", null);
addr.put("city", "Hollywood");
addr.put("state", "CA");
addr.put("zip", "33023");
user.put("address", addr);

JsonNode root = mapper.valueToTree(user);
System.out.println(mapper.writeValueAsString(root));

// prints:
{
  "firstName" : "Harrison",
  "lastName" : "Ford",
  "emailAddress" : [ "harrison@example.com", "hford@actors.com" ],
  "address" : {
    "zip" : "33023",
    "address2" : null,
    "city" : "Hollywood",
    "address1" : "123 Main Street",
    "state" : "CA"
  }
}

And of course, once you have the JsonNode tree model, you can use Jackson data binding to create a Java object (POJO).

User userAgain = mapper.treeToValue(root, User.class);

6. Iterating over a JsonNode

Once you have a JsonNode, how can you find out what it contains? You can loop over its contents and extract the children.

Find the key names of a JsonNode (assuming it is an ObjectNode). And lookup the value JsonNode using JsonNode.get(String).

JsonNode parent = ...;
for (Iterator<String> it = parent.fieldNames() ; it.hasNext() ; ) {
    String field = it.next();
    System.out.println(field + " => " + parent.get(field));
}

Or iterate over the fields, again assuming it is an ObjectNode:

for (Iterator<Map.Entry<String,JsonNode>> it = parent.fields() ;
     it.hasNext() ; ) {
    Map.Entry<String,JsonNode> e = it.next();
    System.out.println(e.getKey() + " => " + e.getValue());
}

Fetch and iterate over all the elements. This one works with an ArrayNode too.

for (Iterator<JsonNode> it = parent.elements() ; it.hasNext() ; ) {
    JsonNode node = it.next();
    System.out.println(node);
}

7. Creating a Tree Model from Scratch

You can also completely generate a tree model from scratch. Start by creating an ObjectNode and populate it using JsonNode.with(). Here we create the above User instance with the associated Address as shown below.

ObjectNode root = mapper.createObjectNode();
root.put("firstName", "Harrison");
root.put("lastName", "Ford");
root.with("address").put("address1", "123 Main Street");
root.with("address").put("address2", null);
root.with("address").put("city", "Hollywood");
root.with("address").put("state", "CA");
root.with("address").put("zip", "33023");
System.out.println(mapper.writeValueAsString(root));

8. Adding Items to a List

For adding lists into the JSON tree model, you can start with the method withArray() and add items as shown:

root.withArray("Genre").add("Drama").add("Horror");

// results in:
{
 ..
 "Genre" : [ "Drama", "Horror" ],
 ..
}

9. Adding from a List

Here is an example of using Java 8 Streams to add nodes from a List:

Arrays.asList("Stephen King (novel)",
                 "Stanley Kubrick (screenplay)",
                 "Diane Johnson (screenplay)")
    .stream()
    .forEach(root.withArray("Writer")::add);

// looks like:
{
 ..
 "Writer" : [ "Stephen King (novel)", "Stanley Kubrick (screenplay)", "Diane Johnson (screenplay)" ]
 ..
}

10. Adding from a Map

And adding to a tree model from a Map is similarly easy:

map.put("cat", "meow");
map.put("dog", "bark");
map.put("cow", "moo");
map
    .entrySet()
    .stream()
    .forEach(e -> root.with("animals").put(e.getKey(), e.getValue()));

// output:
{
 "animals" : {
   "cat" : "meow",
   "cow" : "moo",
   "dog" : "bark"
 }
}

11. Using JsonPointer

JsonPointer is a proposed standard for addressing nodes within a JSON document (somewhat similar to XPath for XML documents). Jackson supports extraction of JsonNodes with a JsonPointer expression using the method JsonNode.at().

JsonNode root = mapper.readTree(new File(jsonFile));
String jsonPtr = ...;
System.out.println(mapper.writeValueAsString(root.at(jsonPtr)));

Let us see some examples of JsonPointer. Consider this JSON.

{
  "firstName" : "Harrison",
  "lastName" : "Ford",
  "emailAddress" : [ "harrison@example.com", "hford@actors.com" ],
  "address" : {
    "zip" : "33023",
    "address2" : null,
    "city" : "Hollywood",
    "address1" : "123 Main Street",
    "state" : "CA"
  }
}

Here are some examples of JsonPointer expression evaluation on this document.

/firstName: "Harrison"

/emailAddress: [ "harrison@example.com", "hford@actors.com" ]

/emailAddress/0: "harrison@example.com"

/emailAddress/1: "hford@actors.com"

/address: {
  "zip" : "33023",
  "address2" : null,
  "city" : "Hollywood",
  "address1" : "123 Main Street",
  "state" : "CA"
}

/address/zip: "33023"

Summary

This article demonstrated how to easily work with the tree model in JSON. We showed how to parse JSON into JsonNode and also how to extract information from the tree model.

Guide to Jackson Annotations

1. Introduction

Jackson provides a number of annotations which help tweak various facets of JSON serialization and deserialization. In this guide, we present some important and commonly used Jackson annotations.

2. @JsonProperty

The annotation @JsonProperty is used to change the JSON property name used in serialization and deserialization. For example,

public class User {
    @JsonProperty("FirstName")
    private String firstName;
}

Output before and after:

// before
...
  "firstName" : "Harrison",
...

// after
  "FirstName" : "Harrison",
...

2.1. Confusion with required

The annotation @JsonProperty defines an attribute required which does not quite work the way it is expected. Despite being specified, it does not mark the property as required either during serialization or deserialization. It applies only when using @JsonProperty with @JsonCreator. For example, the following does not require the presence of the the property “FirstName“:

public class User {
    @JsonProperty(value="LastName", required=true)
    private String lastName;
}

2.2. @JsonProperty and @JsonCreator

@JsonProperty can be used with @JsonCreator to indicate the constructor to use when deserializing JSON. Consider the following class which does not implement a no-arguments default constructor but has a constructor taking appropriate arguments.

public class MyClass {
    private String firstName;
    private String lastName;

    public MyClass(String firstName,String lastName)
    {
	this.firstName = firstName;
	this.lastName = lastName;
    }
}

Create an instance of this class and serialize it to JSON.

MyClass obj = new MyClass("Harrison", "Ford");
System.out.println(mapper.writeValueAsString(obj));

// serialized to:
{
  "firstName" : "Harrison",
  "lastName" : "Ford",
}

Deserializing this JSON results in a JsonMappingException. That is because Jackson cannot create an instance of the class in the absence of a default constructor.

Exception in thread "main" com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of sample.sample11$MyClass: no suitable constructor found, can not deserialize from Object value (missing default constructor or creator, or perhaps need to add/enable type information?)

What if you do not want to add a default constructor due to design considerations? Jackson offers a solution: use @JsonCreator with @JsonProperty to properly mark the constructor. The constructor takes two arguments, both of which are marked required. With this change, Jackson is able to create an instance of MyClass with the proper arguments.

public class MyClass {
    private String firstName;
    private String lastName;
    private Address address;

    @JsonCreator
    public MyClass(@JsonProperty(value = "firstName",required = true) String firstName,
		   @JsonProperty(value = "lastName", required = true) String lastName)
    {
	this.firstName = firstName;
	this.lastName = lastName;
    }
}

3. @JsonIgnore

Property-level annotation used to ignore specified properties. These properties are ignored during both serialization and deserialization. For instance, the following property password is neither written to JSON nor read from JSON.

...
    @JsonIgnore
    private String password;
...

4. @JsonIgnoreProperties

Class level annotation used to ignore multiple properties.

@JsonIgnoreProperties({"bar", "baz"})
public class MyClass
{
    private String foo;
    private String bar;
    private Integer baz;
    ...
}

Additionally, you can use this annotation to ignore unknown properties during deserialization. If unknown properties are present in the JSON, you will get an UnrecognizedPropertyException.

@JsonIgnoreProperties(ignoreUnknown=true, {"bar", "baz"})
public class MyClass
{
    private String foo;
    private String bar;
    ...
}

Parsing the following JSON will not result in an exception due to specifying ignoreUnknown.

{
  "foo" : "value of foo",
  "bar" : "value of bar",
  "qux" : "value of qux"
}

5. @JsonIgnoreType

To ignore a particular class (or built-in type) during serialization and deserialization, use the @JsonIgnoreType annotation on the class. In the following, the MyClass.address property will not be serialized or deserialized.

@JsonIgnoreType
public class Address {
    private String address1;
    ...
}

public class MyClass {
    private Address address;
    private String firstName;
    private String lastName;
}

Note that MyClass.address property has bot been serialized. It will be ignored during deserialization.

{
  "firstName" : "Harrison",
  "lastName" : "Ford"
}

6. @JsonFormat

@JsonFormat is used to specify some extra details when serializing certain types especially dates.

The date field in the following class is serialized by default as a timestamp.

public class MyClass
{
    private Date createdDate = new Date();
}

// serialized to:
{
  "createdDate" : 1485913763410
}

Annotate the date field with @JsonFormat and use pattern to set the date format:

public class MyClass
{
    @JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd@HH:mm:ss.SSSZ")
    private Date createdDate;
}

Serialization now shows the date in the specified format:

{
  "createdDate" : "2017-02-01@01:58:28.130+0000"
}

Deserialization also works as expected. If the date is not specified in the same format as required by the pattern, you get the following exception:

Exception in thread "main" com.fasterxml.jackson.databind.exc.InvalidFormatException: Can not deserialize value of type java.util.Date from String "...": expected format "yyyy-MM-dd@HH:mm:ss.SSSZ" ...

7. @JsonUnwrapped

You can use @JsonUnwrapped to indicate to Jackson that a contained object should be unwrapped during serialization i.e. the contained object properties should be included among the properties of the parent. An example:

public class Address {
    private String address1;
    private String address2;
    private String city;
    private String state;
    private String zip;
}

public class MyClass {
    private String firstName;
    private String lastName;
    @JsonUnwrapped
    private Address address;
}

When an object of MyClass is serialized, the properties of address are included among the properties of MyClass:

{
  "firstName" : "Harrison",
  "lastName" : "Ford",
  "address1" : "123 Main Street",
  "address2" : null,
  "city" : "Hollywood",
  "state" : "CA",
  "zip" : "33023"
}

To prevent clashes between the parent and child object properties, a prefix or suffix can be specified to be added to the child properties:

public class MyClass {
    private String firstName;
    private String lastName;
    @JsonUnwrapped(prefix="addr.")
    private Address address;
}

The resulting JSON is now:

{
  "firstName" : "Harrison",
  "lastName" : "Ford",
  "addr.address1" : "123 Main Street",
  "addr.address2" : null,
  "addr.city" : "Hollywood",
  "addr.state" : "CA",
  "addr.zip" : "33023"
}

During deserialization, either of the forms shown are accepted without errors.

// normal form
{
  "firstName" : "Harrison",
  "lastName" : "Ford",
  "address" : {
    "address1" : "123 Main Street",
    "address2" : null,
    "city" : "Hollywood",
    "state" : "CA",
    "zip" : "33023"
  }
}

// no prefix
{
  "firstName" : "Harrison",
  "lastName" : "Ford",
  "address1" : "123 Main Street",
  "address2" : null,
  "city" : "Hollywood",
  "state" : "CA",
  "zip" : "33023"
}

// with prefix
{
  "firstName" : "Harrison",
  "lastName" : "Ford",
  "addr.address1" : "123 Main Street",
  "addr.address2" : null,
  "addr.city" : "Hollywood",
  "addr.state" : "CA",
  "addr.zip" : "33023"
}

8. @JsonGetter

This annotation can be used on a no-arguments method returning non-void, with the value specified to indicate the name of a logical property. An example, serializing an object of the class below will include a property “joe” defined as the value returned from the method theDate().

public class MyClass {
    private String firstName;
    private String lastName;

    @JsonGetter("joe")
    public Date theDate()
    {
	return new Date();
    }
}

// serialized to:
{
 "firstName" : "Harrison",
 "lastName" : "Ford",
 "joe" : 1485931403261
}

9. @JsonAnyGetter and @JsonAnySetter

In some instances, you may have a situation where a class has dynamic properties stored in a Map and these properties need to be exposed to JSON. In this situation, the getter method or the Map field can be marked with the @JsonAnyGetter annotation. An example will illustrate the point.

public class MyClass {
    private String firstName;
    private String lastName;
    private Address address;
    private Map<String,Object> moreProps = new HashMap<>();
}

// set properties
MyClass obj = new MyClass();
...
obj.setAddress(address);
obj.getMoreProps().put("born", "Chicago, Illinois");
System.out.println(mapper.writeValueAsString(obj));

// serialized to:
{
  "firstName" : "Harrison",
  "lastName" : "Ford",
  "moreProps" : {
    "born" : "Chicago, Illinois"
  }
}

As expected, the Map field appears as a separate object within the container object. Using the @JsonAnyGetter merges the Map entries into the parent JSON.

public class MyClass {
    private String firstName;
    private String lastName;
    private Address address;
    private Map<String,Object> moreProps = new HashMap<>();

    @JsonAnyGetter
    public Map<String,Object> getMoreProps() {
        return moreProps;
    }
}

// create and set object properties here as before
MyClass obj = new MyClass();
...

// serialized to:
{
  "firstName" : "Harrison",
  "lastName" : "Ford",
  "born" : "Chicago, Illinois"
}

That takes care of serialization. How about deserialization? JSON serialized for the above case cannot directly be used for deserialization as it results in an UnrecognizedPropertyException.

The solution is to use @JsonAnySetter on a method which accepts a key and a value to set the property. See below.

public class MyClass {
    ... // as before

    @JsonAnySetter
    public void setMore(String key,Object value)
    {
	moreProps.put(key, value);
    }
}

With this method definition and annotation in place, deserialization is also a breeze.

10. @JsonValue

@JsonValue can be used to serialize the complete state of an object using a single method rather than the usual serialization using properties. An example will help clarify the picture.

The following class marks the toString() method with @JsonValue. When serializing this object, Jackson uses the representation returned from this method as the value of the object. Note that the value returned is not valid JSON.

public class Address {
    private String address1;
    private String address2;
    private String city;
    private String state;
    private String zip;

    @JsonValue
    public String toString()
    {
	StringBuilder sbuf = new StringBuilder();
	sbuf.append(address1).append("|")
	    .append(address2).append("|")
	    .append(city).append("|")
	    .append(state).append("|")
	    .append(zip).append("|");
	return sbuf.toString();
    }
}

Serializing an instance of the above class yields:

"123 Main Street|null|Hollywood|CA|33023|"

Let us see how this works when included as a part of another class.

public class MyClass {
    private String firstName;
    private String lastName;
    private Address address;
}

After suitable initialization and serialization, we have the following JSON output. Note that the method annotated with @JsonValue was used for converting the value of Address to its representation.

{
  "firstName" : "Harrison",
  "lastName" : "Ford",
  "address" : "123 Main Street|null|Hollywood|CA|33023|"
}

10.1. @JsonValue and @JsonCreator

Now, how do we deserialize this JSON into the proper object? We make use of the @JsonCreator annotation to mark a static factory method (or a constructor) which can create the object from its string representation. This method does the opposite of what the @JsonValue method does: it splits the string representation and sets the appropriate fields.

public class Address {
    private String address1;
    private String address2;
    ...

    @JsonCreator
    static public Address fromString(String value)
    {
	String[] values = value.split("\\|");
	Address obj = new Address();
	int index = 0;
	if ( index < values.length ) {
	    obj.setAddress1(values[index]);
	    index++;
	}
	if ( index < values.length ) {
	    obj.setAddress2(values[index]);
	    index++;
	}
	if ( index < values.length ) {
	    obj.setCity(values[index]);
	    index++;
	}
	if ( index < values.length ) {
	    obj.setState(values[index]);
	    index++;
	}
	if ( index < values.length ) {
	    obj.setZip(values[index]);
	    index++;
	}
	return obj;
    }
}

11. @JsonPropertyOrder

Sometimes you may need to specify the order in which the properties are serialized. By default properties are serialized in the order they are defined in the class. If this order is not satisfactory, you can use @JsonPropertyOrder to specify the order of appearance.

The following definition illustrates the point. Note the order in which the properties are serialized in JSON.

public class MyClass {
    private Address address;
    private String firstName;
    private String lastName;
}

// serialized by default as:
{
  "address" : {
    "address1" : "123 Main Street",
    "address2" : null,
    "city" : "Hollywood",
    "state" : "CA",
    "zip" : "33023"
  },
  "firstName" : "Harrison",
  "lastName" : "Ford"
}

Now applying the @JsonPropertyOrder annotation as shown rearranges the order to the required.

@JsonPropertyOrder(value = {"firstName", "lastName", "address"})
static public class MyClass {
    private Address address;
    ...
}

// serialized to:
{
  "firstName" : "Harrison",
  "lastName" : "Ford",
  "address" : {
    "address1" : "123 Main Street",
    "address2" : null,
    "city" : "Hollywood",
    "state" : "CA",
    "zip" : "33023"
  }
}

12. Mixin Annotations

Sometimes the class being used in JSON conversion may not be available for direct modification to add or remove annotations. Maybe the class is a part of a library and hence cannot be edited. How can we declare the annotations that we want on such classes? We use Mixins! Mixin classes can be declared with the desired annotations and added to the ObjectMapper using the addMixIn() method.

Example class:

public class MyClass
{
    private String foo;
    private String bar;
    private Integer baz;
    ...
}

Declare and add a mixin class like this. (The class can be a local class.)

@JsonIgnoreProperties(ignoreUnknown=true)
class Mixin {
    @JsonIgnore
    private Integer baz;
};
mapper.addMixIn(MyClass.class, Mixin.class);

Sample output from serialization. Note that property baz is ignored.

{
  "foo" : "value of foo",
  "bar" : "value of bar"
}

Deserializing this JSON does not cause an exception since ignoreUnknown is set in the mixin.

{
  "foo" : "value of foo",
  "bar" : "value of bar",
  "qux" : "value of qux"
}

Summary

In this article, we looked at some of the most common Jackson annotations for JSON serialization and deserialization.

Using Jackson for JSON Serialization and Deserialization

Learn how to use Jackson to serialize java objects.

1. Introduction

Jackson is one of the most common Java libraries for processing JSON. It is used for reading and writing JSON among other tasks. Using Jackson, you can easily handle automatic conversion from Java objects to JSON and back. In this article, we delve into some common Jackson usage patterns.

2. Converting a POJO to JSON

Suppose we want to convert a sample POJO (Plain Old Java Object) to JSON. An example class is defined below.

public class User {
    private String firstName;
    private String lastName;
    private Date dateOfBirth;
    private List<String> emailAddrs;

    public User(String firstName,String lastName) {
        this.firstName = firstName;
        this.lastName = lastName;
    }

    // standard getters and setters here
}

Converting such an object to JSON is just a couple of lines:

ObjectMapper mapper = new ObjectMapper();
User user = new User("Harrison", "Ford");
user.setEmailAddrs(Arrays.asList("harrison@example.com"));
mapper.writeValue(System.out, user);

// prints:
{"firstName":"Harrison","lastName":"Ford","dateOfBirth":null,"emailAddrs":["harrison@example.com"]}

3. Pretty Printing

Note that the default output is quite compact. Sometimes it is useful to be able to view indented output for debugging purposes. To produce properly indented output, enable the option SerializationFeature.INDENT_OUTPUT on the ObjectMapper instance before using it to serialize the object.

ObjectMapper mapper = new ObjectMapper();
mapper.enable(SerializationFeature.INDENT_OUTPUT);
User user = new User("Harrison", "Ford");
...

The output generated now is nicely indented:

// prints:
{
  "firstName" : "Harrison",
  "lastName" : "Ford",
  "dateOfBirth" : null,
  "emailAddrs" : [ "harrison@example.com" ]
}

4. Ignore NULL fields

How do we tell Jackson not to serialize null values (as for “dateOfBirth” above)? There are a couple of ways. You can tell the ObjectMapper to skip all NULL fields, or you can use annotations to specify per class or per field.

4.1. Configuring ObjectMapper

...
mapper.setSerializationInclusion(Include.NON_NULL);
User user = new User("Harrison", "Ford");
...

// prints:
{
  "firstName" : "Harrison",
  "lastName" : "Ford",
  "emailAddrs" : [ "harrison@example.com" ]
}

4.2. With an Annotation

Use the annotation @JsonInclude(Include.NON_NULL) on the target object class to eliminate serialization of all NULL fields.

@JsonInclude(Include.NON_NULL)
public class User {
    private String firstName;
    private String lastName;
    ...
}

Or on a field (or property) to disable specific NULL fields from being serialized. In the code below, dateOfBirth will be ignored if null, but not heightInM.

...
    @JsonInclude(Include.NON_NULL)
    private Date dateOfBirth;

    private Float heightInM;
...

5. Ignore Empty Arrays

When you have a class with an empty array initializer, the value is serialized as an empty JSON array.

class User {
    ...
    private List<String> phoneNumbers = new ArrayList<>();
    ...
}

// serialized to:
{
  ...
  "phoneNumbers" : [ ],
  ...
}

When you want to skip empty array declarations being output, you can use the following.

mapper.setSerializationInclusion(Include.NON_EMPTY);

6. Generate JSON String

How can we generate a JSON string representation instead of writing it directly to a file or an OutputStream? Maybe you want to store the JSON string in a database. Use the writeValueAsString() method.

// set up user
String jsonStr = mapper.writeValueAsString(user);

7. Formatting Dates

By default, Jackson prints Date fields as numeric timestamps as shown below:

Calendar c = Calendar.getInstance();
c.clear(); c.set(1942, 6, 13);
user.setDateOfBirth(c.getTime());

// prints:
{
  "firstName" : "Harrison",
  "lastName" : "Ford",
  "dateOfBirth" : -866957400000,
  "emailAddrs" : [ "harrison@example.com" ]
}

Turning off numeric timestamp results in serialization of a date in the ISO 8601 format as shown below:

...
mapper.disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);
...

// prints:
{
  "firstName" : "Harrison",
  "lastName" : "Ford",
  "dateOfBirth" : "1942-07-12T18:30:00.000+0000",
  "emailAddrs" : [ "harrison@example.com" ]
}

Change the default date format by using SimpleDateFormat as shown:

...
DateFormat fmt = new SimpleDateFormat("dd-MMM-yyyy");
mapper.disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);
...

// prints:
{
  "firstName" : "Harrison",
  "lastName" : "Ford",
  "dateOfBirth" : "13-Jul-1942",
  "emailAddrs" : [ "harrison@example.com" ]
}

8. Converting JSON to POJO

Reading and converting JSON to a Java object is just as easy with Jackson. Accomplished in just a couple of lines:

ObjectMapper mapper = new ObjectMapper();
User user = mapper.readValue(new File(jsonFile), User.class);

The target class (User in this case) needs a no-arguments default constructor defined – failing which you get this exception:

Exception in thread "main" com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of sample.User: no suitable constructor found, can not deserialize from Object value (missing default constructor or creator, or perhaps need to add/enable type information?)

The situation is easily address by adding a no-arguments default constructor.

public class User {
    private String firstName;
    private String lastName;
    private Date dateOfBirth;
    private List<String> emailAddrs;

    public User() {}
    ...
}

9. Adjusting Date Format

As before, the date format needs to be adjusted unless it is in ISO 8601 format. Parsing the following JSON fails:

{
  "firstName" : "Harrison",
  "lastName" : "Ford",
  "dateOfBirth" : "13-Jul-1942",
  "emailAddrs" : [ "harrison@example.com", null ],
}

The following exception is reported:

Exception in thread "main" com.fasterxml.jackson.databind.exc.InvalidFormatException: Can not deserialize value of type java.util.Date from String "13-Jul-1942": not a valid representation (error: Failed to parse Date value '13-Jul-1942': Can not parse date "13-Jul-1942": not compatible with any of standard forms ("yyyy-MM-dd'T'HH:mm:ss.SSSZ", "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", "EEE, dd MMM yyyy HH:mm:ss zzz", "yyyy-MM-dd"))

Specify the date format if you are using a non-standard format as shown:

DateFormat fmt = new SimpleDateFormat("dd-MMM-yyyy");
mapper.setDateFormat(fmt);

10. Ignoring Unknown Properties

Sometimes the JSON you are trying to read might include some properties not defined in the Java class. Maybe the JSON was updated but the change is not yet reflected in the Java class. In such cases, you might end up with an exception like this:

Exception in thread "main" com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "gender" (class sample.User), not marked as ignorable (6 known properties: ...)

You can tell Jackson to ignore such properties on a global level by disabling a deserialization feature as follows:

mapper.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES);

Alternatively, you can use the following annotation on the class to ignore unknown properties.

@JsonIgnoreProperties(ignoreUnknown=true)
public class User {
    ...
}

11. Serializing Array of Objects to JSON

Serializing an array of objects to JSON is straightforward.

ObjectMapper mapper = new ObjectMapper();
List<User> users = new ArrayList<>();
users.add(new User(...));
users.add(new User(...));
System.out.println(mapper.writeValueAsString(users));

The array is properly serialized as shown:

[ {
  "firstName" : "Harrison",
  "lastName" : "Ford",
  "dateOfBirth" : "13-Jul-1942",
  ...},
  {
  "firstName" : "Samuel",
  "lastName" : "Jackson",
  "dateOfBirth" : "21-Dec-1948",
  ...} ]

12. Deserialize Array of Objects from JSON

Several methods are available to deserialize a JSON array to a collection of Java objects. Use whatever method suits you depending on your requirements.

Deserializing a JSON array to a Java array of objects of a particular class is as simple as:

User[] users = mapper.readValue(new File(jsonFile), User[].class);
System.out.println(mapper.writeValueAsString(users));

Using a JavaType is useful when constructing collections or parametric types.

JavaType listType = mapper
    .getTypeFactory()
    .constructCollectionType(List.class, User.class);
List<User> users = mapper.readValue(new File(jsonFile), listType);
System.out.println(mapper.writeValueAsString(users));

A third method is to create and use a TypeReference.

TypeReference ref = new TypeReference<List<User>>(){};
List<User> users = mapper.readValue(new File(jsonFile), ref);
System.out.println(mapper.writeValueAsString(users));

Summary

This article showed you how to convert Java objects to JSON and back. We also covered a few common use cases.

Java Blocking Queue

1. Introduction

Java provides the BlockingQueue, a Queue which allows waiting for an element to become available for retrieval. It also allows waiting for a slot to become available when storing elements. A BlockingQueue implementation is thread-safe so one thread can attempt to store an element while a different thread can attempt to remove an element.

2. Creating a BlockingQueue

The ArrayBlockingQueue is an implementation of a bounded BlockingQueue backed by an array. The size of the backing array can be specified while creating the BlockingQueue. This refers to the number of elements that can be stored in the queue before the next put() operation results in the thread being put to sleep. Once this happens, elements must be removed from the BlockingQueue before storing any more.

BlockingQueue<String> queue = new ArrayBlockingQueue<>(2);

3. Worker Thread Pool for Processing

One possible use of a BlockingQueue is as an application worker pool. In this scenario, some threads post tasks onto the queue and other threads can pick up the tasks and complete it. We will use an ExecutorService to setup the thread pool.

ExecutorService esvc = Executors.newFixedThreadPool(2);

4. Creating Producers and Consumers

A Producer is an object which produces a task to be executed. The task details are added to the BoundedQueue and is picked up by a Consumer when it is ready to process the task. Using a BoundedQueue for a scenario like this (the producer-consumer problem) means the producer goes to sleep if the queue is full and the consumer sleeps if the queue is empty.

The following is a producer task formulated as a lambda. It tries to store a task (“apple”) into the queue declared above. After the put() succeeds, it returns a status.

Callable<String> producer = () -> {
    String s = "apple";
    queue.put(s);
    return "put -> " + s;
}

And here is the consumer task which attempts to retrieve the next task and process it. Once the processing is done, it returns a status.

Callable<String> consumer = () -> {
    String s = queue.take();
    // consume task here
    return "take -> " + s;
}

In the code below, we declare an array of tasks and submit it to the thread pool. In your code, you do not have to add producers and consumers together as shown.

List<Callable<String>> elements =
    Arrays.asList(() -> {
            String s = "apple";
            queue.put(s);
            return "put -> " + s;
        },
        () -> {
            String s = "orange";
            queue.put(s);
            return "put -> " + s;
        },
        () -> {
            String s = queue.take();
            return "take -> " + s;
        },
        () -> {
            String s = "banana";
            queue.put(s);
            return "put -> " + s;
        },
        () -> {
            String s = queue.take();
            return "take -> " + s;
        },
        () -> {
            String s = queue.take();
            return "take -> " + s;
        });

5. Adding a Shutdown Task

We also add a “shutdown” task at the end to shutdown the thread pool properly after all the tasks complete.

Callable<String> shutdownTask = () -> {
            esvc.shutdown();
            return "shutdown";
        });

Let us now submit the tasks for execution. At this point, the thread pool executes kicks off the tasks using the available threads (2 threads as shown above).

esvc.invokeAll(elements);

6. Waiting for Completion

The invokeAll() method returns a list of Future<String> which we can process using the streams facility to wait for the tasks to complete. The method Future<String>.get() waits for completion of each task.

esvc.invokeAll(elements)
    .forEach(f -> {
            try {
                System.out.println(f.get());
            } catch(Exception ex) {
                System.out.println("get failed: " + ex.getMessage());
            }
        });

The output is shown below. The producer tasks insert the task into the queue and the consumer tasks retrieve the task and process it. Finally the shutdown task closes the thread pool.

put -> apple
put -> orange
take -> apple
put -> banana
take -> banana
take -> orange
shutdown

7. Deadlock

A deadlock might occur in the above case if we use the newSingleThreadExecutor() method which executes the tasks in sequence. If the BoundingQueue is created with less capacity, the producers might be put to sleep waiting for storage. If the consumers fail to clear the queue fast enough, the producers might be left waiting for a longer time. It is important to keep track of such issues and work around them. However, using newFixedThreadPool() to create the ExecutorService spawns more threads and runs the consumers so the producers have space for storage.

Summary

In this article, we saw how to use the BoundedQueue to create a producer-consumer system. In such a system, producers create tasks which are completed in another thread by consumers. Invoking Future<?>.get() waits for the tasks to complete and deliver the status.

Java Concurrency and Executors

1. Introduction

Java provides a package java.util.concurrent which includes many facilities to ease concurrent programming. In this article, we take a look at java threads and Executors – what they are and how they work, etc.

2. Runnable Task as a Lambda

Since the early days of Java 1.0, Java provides a Runnable interface which must be implemented to run a task in a separate thread. An sample implementation of a Runnable task is shown below. Note that rather than creating a class implementing the Runnable interface, the code creates a Runnable lambda. The task is then executed in the main thread as well as a second thread.

static private void example_1(String[] args) throws Exception
{
    Runnable task = () -> {
	String threadName = Thread.currentThread().getName();
	System.out.println("Hello " + threadName);
    };

    task.run();

    Thread thread = new Thread(task);
    thread.start();
    System.out.println("Done.");
}

The output is shown below. Note that the main thread completes before the second thread but waits for it to complete before the program terminates.

Hello main
Done.
Hello Thread-0

3. Creating an Executor

With the new Executor Framework in Java 8, java provides an Executor interface with a single method execute(Runnable) for executing a Runnable task. As opposed to creating a Thread with a Runnable (as shown above), you can create an Executor and run one or more tasks as follows:

Executor executor = ...;
executor.execute(task1);
executor.execute(task2);

The advantage of this approach is that the Executor implementation takes care of thread creation and management. To this end, java provides several implementations of this interface supporting various techniques such as executing tasks in a thread pool, executing tasks sequentially in a worker thread, etc.

In addition to the Executor interface, Java also provides an ExecutorService which is a sub-interface of Executor. This interface provides additional facilities over Executor including convenience methods to execute multiple tasks, submit tasks for execution, etc.

Executors is a convenience class providing factory methods to create various kinds of ExecutorService implementation objects. The method newSingleThreadExecutor() creates a thread and executes pending tasks sequentially. A task is submitted for execution using the submit() method.

static private void example_2(String[] args) throws Exception
{
    ExecutorService esvc = Executors.newSingleThreadExecutor();
    Runnable task = () -> {
	try {
	    String threadName = Thread.currentThread().getName();
	    System.out.println("Thread " + threadName + " started");
	    TimeUnit.SECONDS.sleep(2);
	    System.out.println("Thread " + threadName + " ended");
	} catch(InterruptedException ex) {
	    System.err.println("Task interrupted.");
	}
    };

    esvc.submit(task);
}

On running this example, the ExecutorService executes the submitted task and continues to wait for more tasks without exiting. This is the default for all ExecutorService objects. Hit Control-C to halt the program.

4. Serial Execution of Tasks

In addition to executing a Runnable, an ExecutorService can also run a Callable<?>. The following class implements Callable<String> which returns a Future<String> after the execution is completed. It can be scheduled for execution using an ExecutorService.

static private class DelayTask implements Callable<String>
{
    private String name;
    private int secsDelay;

    public DelayTask(String name,int secsDelay) {
	this.name = name;
	this.secsDelay = secsDelay;
    }

    @Override
	public String call() {
	System.out.println(name + " started");
	try { TimeUnit.SECONDS.sleep(secsDelay); }
	catch(InterruptedException ex) {
	    System.err.println(name + ": interrupted");
	}
	System.out.println(name + " ended");
	return name;
    }
}

Let us create a bunch of these tasks and execute them using an ExecutorService created from newSingleThreadExecutor(). This ExecutorService executes the tasks sequentially. At the end, we add another task to shutdown the ExecutorService. Due to Future<?>.get(), the code waits for all the tasks to terminate and then cleanly shuts down the ExecutorService which cleans up all the resources.

static private void example_3(String[] args) throws Exception
{
    ExecutorService esvc = Executors.newSingleThreadExecutor();
    List<Callable<String>> tasks =
	Arrays.asList(new DelayTask("task 1", 2),
		      new DelayTask("task 2", 3),
		      new DelayTask("task 3", 1),
		      () -> {
			  esvc.shutdown();
			  return "shutdown";
		      });
    esvc.invokeAll(tasks)
	.stream()
	.map(future -> {
		try { return future.get(); }
		catch(Exception ex) {
		    return "exception: " + ex.getMessage();
		}
	    })
	.forEach(System.out::println);
}

Here is the output from the code above. Notice that tasks are executed one after another followed by the shutdown task. This is because we used the newSingleThreadExecutor() method which creates an ExecutorService with that characteristic.

task 1 started
task 1 ended
task 2 started
task 2 ended
task 3 started
task 3 ended
task 1
task 2
task 3
shutdown

5. Executing Tasks with a Thread Pool

Let us now examine how a thread pool returned from newCachedThreadPool() behaves. The source code is shown below; it uses the DelayTask class defined above.

static private void example_4(String[] args) throws Exception
{
    ExecutorService esvc = Executors.newCachedThreadPool();
    List<Callable<String>> tasks =
	Arrays.asList(new DelayTask("task 1", 2),
		      new DelayTask("task 2", 3),
		      new DelayTask("task 3", 10),
		      () -> {
			  System.err.println("Requesting shutdown ..");
			  esvc.shutdown();
			  return "shutdown";
		      });
    esvc.invokeAll(tasks)
	.stream()
	.map(future -> {
		try { return future.get(); }
		catch(Exception ex) {
		    return "exception: " + ex.getMessage();
		}
	    })
	.forEach(System.out::println);
}

Here is the output from this code. Notice that a shutdown is requested of the ExecutorService after the tasks have been scheduled. The ExecutorService allows the tasks to complete execution before shutting down. It does not, however, allow any more tasks to be scheduled after shutdown() has been invoked.

Task task 1 started
Task task 2 started
Task task 3 started
Requesting shutdown ..
Task task 1 ended
Task task 2 ended
Task task 3 ended
task 1
task 2
task 3
shutdown

6. Shutdown Thread Pool Immediately

When it is necessary to shutdown the thread pool immediately as opposed to waiting for all tasks to complete, we can use the shutdownNow() method.

static private void example_5(String[] args) throws Exception
{
    ExecutorService esvc = Executors.newCachedThreadPool();
    List<Callable<String>> tasks =
	Arrays.asList(new DelayTask("task 1", 2),
		      new DelayTask("task 2", 3),
		      new DelayTask("task 3", 10),
		      () -> {
			  System.err.println("Requesting shutdown ..");
			  esvc.shutdownNow();
			  return "shutdown";
		      });
    esvc.invokeAll(tasks)
	.stream()
	.map(future -> {
		try { return future.get(); }
		catch(Exception ex) {
		    return "exception: " + ex.getMessage();
		}
	    })
	.forEach(System.out::println);
}

As the output below shows, the tasks that have not yet been completed are interrupted and the thread pool is terminated.

Task task 1 started
Task task 2 started
Task task 3 started
Requesting shutdown ..
task 3: interrupted
Task task 3 ended
task 1: interrupted
Task task 1 ended
task 2: interrupted
Task task 2 ended
task 1
task 2
task 3
shutdown

Summary

This article provided an introduction to threading in Java as well as the new Executor framework in Java 8. The Executor framework simplifies a lot of plumbing code related to thread creation and management. It also brings new capabilities to threading in Java including thread pools, worker threads which can schedule tasks sequentially. As the code above demonstrated, it is also possible to implement code to shutdown the thread pool cleanly after all the tasks have completed.

How to Generate RSA Keys in Java

Learn how to generate RSA keys and digitally sign files in java.

1. Introduction

Let us learn the basics of generating and using RSA keys in Java.

Java provides classes for the generation of RSA public and private key pairs with the package java.security. You can use RSA keys pairs in public key cryptography.

Public key cryptography uses a pair of keys for encryption. Distribute the public key to whoever needs it but safely secure the private key.

Public key cryptography can be used in two modes:

Encryption: Only the private key can decrypt the data encrypted with the public key.

Authentication: Data encrypted with the private key can only be decrypted with the public key thus proving who the data came from.

2. Generating a Key Pair

First step in creating an RSA Key Pair is to create a KeyPairGenerator from a factory method by specifying the algorithm (“RSA” in this instance):

KeyPairGenerator kpg = KeyPairGenerator.getInstance("RSA");

Initialize the KeyPairGenerator with the key size. Use a key size of 1024 or 2048. Currently recommended key size for SSL certificates used in e-commerce is 2048 so that is what we use here.

kpg.initialize(2048);
KeyPair kp = kpg.generateKeyPair();

From the KeyPair object, get the public key using getPublic() and the private key using getPrivate().

Key pub = kp.getPublic();
Key pvt = kp.getPrivate();

3. Saving the Keys in Binary Format

Save the keys to hard disk once they are obtained. This allows re-using the keys for encryption, decryption and authentication.

String outFile = ...;
out = new FileOutputStream(outFile + ".key");
out.write(pvt.getEncoded());
out.close();

out = new FileOutputStream(outFile + ".pub");
out.write(pvt.getEncoded());
out.close();

What is the format of the saved files? The key information is encoded in different formats for different types of keys. Here is how you can find what format the key was saved in. On my machine, the private key was saved in PKCS#8 format and the public key in X.509 format. We need this information below to load the keys.

System.err.println("Private key format: " + pvt.getFormat());
// prints "Private key format: PKCS#8" on my machine

System.err.println("Public key format: " + pub.getFormat());
// prints "Public key format: X.509" on my machine

3.1. Load Private Key from File

After saving the private key to a file (or a database), you might need to load it at a later time. You can do that using the following code. Note that you need to know what format the data was saved in: PKCS#8 in our case.

/* Read all bytes from the private key file */
Path path = Paths.get(keyFile);
byte[] bytes = Files.readAllBytes(path);

/* Generate private key. */
PKCS8EncodedKeySpec ks = new PKCS8EncodedKeySpec(bytes);
KeyFactory kf = KeyFactory.getInstance("RSA");
PrivateKey pvt = kf.generatePrivate(ks);

3.2 Load Public Key from File

Load the public key from a file as follows. The public key has been saved in X.509 format so we use the X509EncodedKeySpec class to convert it.

/* Read all the public key bytes */
Path path = Paths.get(keyFile);
byte[] bytes = Files.readAllBytes(path);

/* Generate public key. */
X509EncodedKeySpec ks = new X509EncodedKeySpec(bytes);
KeyFactory kf = KeyFactory.getInstance("RSA");
PublicKey pub = kf.generatePublic(ks);

4. Use Base64 for Saving Keys as Text

Save the keys in text format by encoding the data in Base64. Java 8 provides a Base64 class which can be used for the purpose. Save the private key with a comment as follows:

Base64.Encoder encoder = Base64.getEncoder();

String outFile = ...;
Writer out = new FileWriter(outFile + ".key");
out.write("-----BEGIN RSA PRIVATE KEY-----\n");
out.write(encoder.encodeToString(pvt.getEncoded()));
out.write("\n-----END RSA PRIVATE KEY-----\n");
out.close();

And the public key too (with a comment):

out = new FileWriter(outFile + ".pub");
out.write("-----BEGIN RSA PUBLIC KEY-----\n");
out.write(encoder.encodeToString(kp.getPublic()));
out.write("\n-----END RSA PUBLIC KEY-----\n");
out.close();

5. Generating a Digital Signature

As mentioned above, one of the purposes of public key cryptography is digital signature i.e. you generate a digital signature from a file contents, sign it with your private key and send the signature along with the file. The recipient can then use your public key to verify that the signature matches the file contents.

Here is how you can do it. Use the signature algorithm “SHA256withRSA” which is guaranteed to be supported on all JVMs. Use the private key (either generated or load from file as shown above) to initialize the Signature object for signing. It is then updated with contents from the data file and the signature is generated and written to the output file. This output file contains the digital signature and must be sent to the recipient for verification.

Signature sign = Signature.getInstance("SHA256withRSA");
sign.initSign(pvt);

InputStream in = null;
try {
    in = new FileInputStream(dataFile);
    byte[] buf = new byte[2048];
    int len;
    while ((len = in.read(buf)) != -1) {
    sign.update(buf, 0, len);
    }
} finally {
    if ( in != null ) in.close();
}

OutputStream out = null;
try {
    out = new FileOutputStream(signFile);
    byte[] signature = sign.sign();
    out.write(signature);
} finally {
    if ( out != null ) out.close();
}

6. Verifying the Digital Signature

The recipient uses the digital signature sent with a data file to verify that the data file has not been tampered with. It requires access to the sender’s public key and can be loaded from a file if necessary as presented above.

The code below updates the Signature object with data from the data file. It then loads the signature from file and uses Signature.verify() to check if the signature is valid.

Signature sign = Signature.getInstance("SHA256withRSA");
sign.initVerify(pub);

InputStream in = null;
try {
    in = new FileInputStream(dataFile);
    byte[] buf = new byte[2048];
    int len;
    while ((len = in.read(buf)) != -1) {
    sign.update(buf, 0, len);
    }
} finally {
    if ( in != null ) in.close();
}

/* Read the signature bytes from file */
path = Paths.get(signFile);
bytes = Files.readAllBytes(path);
System.out.println(dataFile + ": Signature " +
   (sign.verify(bytes) ? "OK" : "Not OK"));

And that in a nutshell is how you can use RSA public and private keys for digital signature and verification.

Source Code

Go here for the source code.

Java – Read File Line by Line Using Java 8 Streams

1. Introduction

Java 8 Streams provide a cool facility to apply functional-style operations on Collection-based classes such as List and Map. These functional-style operations are very expressive and allow elimination of much boiler-plate code for processing pipelines which operate on  elements in these collections.

2. A Streams Example

An example is shown below. Here we process a list of airport names as follows:

1. Select airports that start with “B”

2. Convert the airport name to uppercase

3. Sort the list

4. And print out the element

List<String> airports =
    Arrays.asList("Birmingham-Shuttlesworth International",
		  "Anchorage International",
		  "Deadhorse",
		  "Phoenix Sky Harbor International",
		  "Tucson International",
		  "Los Angeles International",
		  "San Francisco International",
		  "Burbank Bob Hope Airport",
		  "Long Beach Airport",
		  "Oakland International");

airports
    .stream()
    .filter(a -> a.startsWith("B"))
    .map(String::toUpperCase)
    .sorted()
    .forEach(System.out::println);

// prints the following
// BIRMINGHAM-SHUTTLESWORTH INTERNATIONAL
// BURBANK BOB HOPE AIRPORT

Wouldn’t it be nice to apply these processing pipelines to other sequences such as the lines of a file? In this article, we show how to do exactly that.

4. Reading a File Line By Line

To read a file line by line in Java, we use a BufferedReader instance and read in a loop till all the lines are exhausted.

try (BufferedReader in = new BufferedReader(new FileReader(textFile))) {
    String line;
    while ((line = in.readLine()) != null) {
        // process line here
    }
}

3. Implement a Spliterator Class

To turn a BufferedReader into a class capable of being used with the Java 8 Streams API, we need to provide an implementation of the Spliterator interface. Shown is the LineReaderSpliterator class which implements Spliterator<String> and turns a BufferedReader into a stream of lines.

public class LineReaderSpliterator implements Spliterator<String>
{
    private final BufferedReader reader;
    private java.io.IOException exception;

    public LineReaderSpliterator(BufferedReader reader) {
	this.reader = reader;
    }

    public java.io.IOException ioException() { return exception; }

    public int characteristics() {
	return DISTINCT | NONNULL | IMMUTABLE;
    }

    public long estimateSize() {
	return Long.MAX_VALUE;
    }

    public boolean tryAdvance(Consumer<? super String> action) {
	try {
	    String line = reader.readLine();
	    if ( line != null ) {
		action.accept(line);
		return true;
	    } else return false;
	} catch(java.io.IOException ex) {
	    this.exception = ex;
	    return false;
	}
    }

    public Spliterator<String> trySplit() { return null; }
}

3.1 Constructor

The LineReaderSpliterator is initialized with an instance of BufferedReader which serves as the input line source.

public LineReaderSpliterator(BufferedReader reader) {
    this.reader = reader;
}

3.2 Characteristics

The characteristics of the Spliterator must be indicated with the implementation of the characteristics() method. The result must be an OR-ed values from the following:

ORDERED: indicates that the order of elements is defined. The ordering is expected to be preserved in parallel computations.

DISTINCT: Each element is distinct from another element.

SORTED: The sequence is sorted. In our case, the lines may not be sorted so this bit is not set.

SIZED: This bit must be set to indicate that the estimate of size returned by estimateSize() is correct. For our case, we do not know the number of lines in a file so this bit is not set.

NONNULL: Elements are guaranteed to be non-null.

IMMUTABLE: Requires that element source should not be modified to add, replace or remove elements.

CONCURRENT: Indicates that element source can be modified concurrently with additions, replacements and removals from multiple threads.

SUBSIZED: If the spliterator can be split and child spliterators are SIZED and SUBSIZED.

In our case, the spliterator is specified as DISTINCT, NONNULL and IMMUTABLE.

3.3 Size estimation

Our spliterator does not know the size of the collection since the number of lines in the BufferedReader is not known. If the size is now known or is unbounded, the method must return Long.MAX_VALUE.

public long estimateSize() {
    return Long.MAX_VALUE;
}

3.4 Process Next Element

The method to process the next element is the tryAdvance() method which accepts a functional interface Consumer<? super T>. Our implementation attempts to read a line and if successful (EOF not reached) invokes action.accept(). If EOF is reached or an exception occurs, the method return false to indicate end-of-sequence.

public boolean tryAdvance(Consumer<? super String> action) {
    try {
	String line = reader.readLine();
	if ( line != null ) {
	    action.accept(line);
	    return true;
	} else return false;
    } catch(java.io.IOException ex) {
	this.exception = ex;
	return false;
    }
}

3.5 Can the Spliterator split?

If the spliterator can partitioned to return separate ranges of elements, a new Spliterator must be returned. We cannot partition the input into separate sequences so we return null.

public Spliterator<String> trySplit() { return null; }

 4. Using the Stream

We can now use the Spliterator implementation to convert a BufferedReader into a Java 8 stream as follows:

static private Stream<String> createStreamReader(BufferedReader reader)
{
    LineReaderSpliterator s = new LineReaderSpliterator(reader);
    return StreamSupport.stream(s, false);
}

The earlier streams example can now be written as shown.

BufferedReader reader = null;
try {
    reader = new BufferedReader(new FileReader(textFile));
    createStreamReader(reader)
	.filter(a -> a.startsWith("B"))
	.map(String::toUpperCase)
	.sorted()
	.forEach(System.out::println);
} finally {
    if ( reader != null ) reader.close();
}

Check the output shown below:

BALTIMORE-WASHINGTON INTERNATIONAL
BANGOR INTERNATIONAL
BIRMINGHAM-SHUTTLESWORTH INTERNATIONAL
BRADLEY INTERNATIONAL
BURBANK BOB HOPE AIRPORT
BURLINGTON INTERNATIONAL

The example does not store the names in a data structure. Rather, names are directly read from the file and the processing pipeline is applied to the sequence of elements.

Summary

We have demonstrated how to read lines from a file and process it using Java 8 streams. This requires implementation of a Spliterator class for delivering a “stream” view of any sequence. The advantage of such an approach is the ease of filtering and processing text files.

How to Modify XML File in Java

1. Introduction

Let us learn how to modify an XML file to remove unwanted information.

One method to remove XML nodes is to use the XML DOM Api to search the XML structure and remove unwanted nodes. While this sounds easy, using the DOM Api is quite hard especially for anything more than trivial searches as this article demonstrates.

An easier method to navigate and remove unwanted Nodes is to use XPath. Even complex search and removal is quite easy as we shall see.

See this article for details on parsing an XML file to obtain the XML Document.

2. Using removeChild() to remove Nodes

Once a particular node is identified for removal, it can be removed quite easily by invoking removeChild() on the parent Node.

static private void removeNode(Node node)
{
  Node parent = node.getParentNode();
  if ( parent != null ) parent.removeChild(node);
}

2.1 Saving the Modified XML Document

After the required modifications are done, the XML Document can be saved by using a Transformer.

Initialize the Transformer as shown:

tform = TransformerFactory.newInstance().newTransformer();
tform.setOutputProperty(OutputKeys.INDENT, "yes");
tform.setOutputProperty("{http://xml.apache.org/xslt}indent-amount", "2");

Save the modified XML document quite easily using the transformer instance.

tform.transform(new DOMSource(document), new StreamResult(System.out));

3. Searching the XML Document

The XML data set we are using is the TSA airport and checkpoint data available here. We would like to search this data set for the airport in Mobile, AL (identified as <shortcode>MOB</shortcode> in the data set). The following code checks each node whether it matches the query.

static private boolean testNode(Node node)
{
    NodeList nlist = node.getChildNodes();
    for (int i = 0 ; i < nlist.getLength() ; i++) {
	Node n = nlist.item(i);
	String name = n.getLocalName();
	if ( name != null && name.equals("shortcode") ) {
	    return n.getTextContent().equals("MOB");
	}
    }
    return false;
}

Collect the nodes to be removed by searching from the document root.

List<Node> nodes = new ArrayList<>();
NodeList nlist = document.getFirstChild().getChildNodes();
for (int i = 0 ; i < nlist.getLength() ; i++) {
    Node node = nlist.item(i);
    if ( testNode(node) ) nodes.add(node);
}

As you can see from the implementation of testNode(), complex XML search is hard using just the DOM API.

4. Using XPath to Find and Remove Nodes

XPath can be used to easily query for nodes within an XML document.

An initial setup process is required for using XPath to search.

XPathFactory xfact = XPathFactory.newInstance();
XPath xpath = xfact.newXPath();

Here is a method to query for nodes and remove them from the document.

static private void queryRemoveNodes(String xpathStr)
{
Object res = xpath.evaluate(xpathStr, document, PathConstants.NODESET);
NodeList nlist = (NodeList)res;
for (int i = 0 ; i < nlist.getLength() ; i++) {
    Node node = nlist.item(i);
    Node parent = node.getParentNode();
    if ( parent != null ) parent.removeChild(node);
}
}

The previous example to remove the airport for Mobile, AL is written as:

queryRemoveNode("/airports/airport[shortcode = 'MOB']");

The removed node is:

<airport>
  <name>Mobile Regional</name>
  <shortcode>MOB</shortcode>
  <city>Mobile</city>
  <state>AL</state>
  <latitude>30.6813</latitude>
  <longitude>-88.2443</longitude>
  <utc>-6</utc>
  <dst>True</dst>
  <precheck>true</precheck>
  <checkpoints>
    <checkpoint>
      <id>1</id>
      <longname>MOB-A</longname>
      <shortname>MOB-A</shortname>
    </checkpoint>
  </checkpoints>
</airport>

Furthermore, to remove just the <checkpoints> element from the above node, use the following:

queryRemoveNodes("/airports/airport[shortcode = "MOB"]/checkpoints");

Easily remove a bunch of nodes matching an expression.

queryRemoveNodes("/airports/airport[latitude < 20]");

Summary

There are two ways of removing nodes from an XML document. The direct method is to search for nodes using the DOM Api and remove them. An easier way is to use XPath to query and remove the nodes matching even complex queries.

How to Extract Data from XML in Java

1. Introduction

In a previous article, we looked into parsing an XML file and converting it to DOM (Document Object Model). The XML DOM object itself is not very useful in an application unless it can be used to extract required data. In this article, let us see how to extract data from XML in Java.

We demonstrate two approaches to extracting data from the XML document. One is a straightforward navigation of the DOM structure to extract fragments of data. Another way is to use XPath to describe and extract the exact information needed with an expression.

2. Accessing the XML Root Element

The most commonly used class in the DOM API is the Node class. All other types of XML artifacts are represented as a Node. These include elements, attributes, text within elements, CDATA, etc.

The most common type of Node we will be concerned with is the element. An element node has attributes, zero or more child elements, text nodes, etc.

A Document is a special type of Node which is obtained as a result of parsing the XML. Use the getFirstChild() method of a Document to get the XML root element.

Node rootElement = document.getFirstChild();

3. Accessing XML Element Children

Access the list of children of an element with the getChildNodes() method. A list of child nodes including elements, text nodes, CDATA, comments, etc are returned. It can be processed like this:

NodeList nlist = node.getChildNodes();
for (int i = 0 ; i < nlist.getLength() ; i++) {
    Node child = nlist.item(i);
    // process the child node here
}

A child element and its text contents can be checked as follows: A child element <shortcode>PBI</shortcode> is selected for processing here.

String name = child.getLocalName();
if ( name != null && name.equals("shortcode") ) {
    if ( child.getTextContent().equals("PBI") ) {
        // process element here
    }
}

4. Generating XML Output

Print the whole XML fragment from a node once it is selected. This includes all the child nodes, text, attributes, etc.

Create a Transformer object from the factory object:

Transformer tform = TransformerFactory.newInstance().newTransformer();

Pretty-printing the XML helps in visualizing the structure. You can enable pretty-printing as shown. Here an indentation of 2 spaces is being specified.

tform.setOutputProperty(OutputKeys.INDENT, "yes");
tform.setOutputProperty("{http://xml.apache.org/xslt}indent-amount", "2");

And generate the XML output from a Node object for printing:

tform.transform(new DOMSource(node), new StreamResult(System.out));

5. A More Complex Example

Let us look at a more complex example of XML data extraction with some real-world data. The XML data set we are using is the publicly available TSA airport and checkpoint data available here (warning: large file download). This data includes airport information including GPS coordinates and checkpoints.

Let us search this XML data set for information within specified GPS coordinates: locate airports within latitudes range of (25, 30), longitude range of (-90, -80). We search for matching nodes from the root node of the XML.

List<Node> res = new ArrayList<>();
NodeList nlist = rootNode.getChildNodes();
for (int i = 0 ; i < nlist.getLength() ; i++) {
    Node node = nlist.item(i);
    NodeList children = node.getChildNodes();
    boolean foundLat = false, foundLong = false;
    for (int j = 0 ; j < children.getLength() ; j++) {
	Node child = children.item(j);
	String name = child.getLocalName();
	if ( name == null ) continue;
	if ( name.equals("latitude") ) {
	    float lat = Float.parseFloat(child.getTextContent());
	    if ( lat > 25 && lat < 30 ) foundLat = true;
	} else if ( name.equals("longitude") ) {
	    float lng = Float.parseFloat(child.getTextContent());
	    if ( lng > -90 && lng < -80 ) foundLong = true;
	}
    }
    if ( foundLat && foundLong ) res.add(node);
}

The code above loops through all elements under the root node and selects those children which match the specified conditions: latitude between (25, 30) and longitude between (-90, -80).

As you can see, the code is quite complex and prone to errors. And this is just for finding nodes for some rather simple conditions.

6. Using XPath to Extract Information

Java provides an XPath API which can be used in conjunction with the XML DOM to extract information from XML in an easy manner. XPath in initialized with the application as follows:

XPathFactory xfact = XPathFactory.newInstance();
XPath xpath = xfact.newXPath();

To extract possibly multiple nodes which match an XPath expression, the following method can be used.

Object res = xpath.evaluate(xpathStr, document, XPathConstants.NODESET);

If you know that a single node will match the expression, you can use this method instead.

Object res = xpath.evaluate(xpathStr, document, XPathConstants.NODE);

Maybe you are trying to extract application configuration information from XML? In that case, you might prefer fetching String values in a single call.

String value = xpath.evaluate(xpathStr, document);

7. Some Examples

To compare with the earlier examples, let us find the airport node where <shortcode> equals “PBI“:

String xpathStr = "/airports/airport[shortcode = 'PBI']";
Object res = xpath.evaluate(xpathStr, document, XPathConstants.NODESET);
show((NodeList)res);

Results in the output shown below (partially):

...
  <shortcode>PBI</shortcode>
  <city>West Palm Beach</city>
  <state>FL</state>
  <latitude>26.6831606</latitude>
  <longitude>-80.0955892</longitude>
  <utc>-5</utc>
  <dst>True</dst>
...

And here is the second example: find airports with latitude between (25, 30) and longitude between (-90, -80).

String xpathStr = "/airports/airport[latitude > 25 and latitude < 30 and longitude > -90 and longitude < -80]";
Object res = xpath.evaluate(xpathStr, document, XPathConstants.NODESET);
show((NodeList)res);

Summary

This article demonstrated a couple of ways of extracting data from XML documents. A direct way is to navigate the DOM structure and perform the extraction. This is error prone and sensitive to changes in XML structure. An easier way is to use XPath expression search to extract required information.