Learn how to use Jackson Streaming to convert a large CSV to JSON.
“We are a way for the cosmos to know itself.”
― Carl Sagan, Cosmos
Let us today look into converting a large CSV to JSON without running into memory issues. This previous article showed how to parse CSV and output the data to JSON using Jackson. However, since that code loads the entire data into memory, it will run into issues loading large CSV files such as:
Implement a Pivot Table in Java using Java 8 Streams and Collections.
“Money may not buy happiness, but I’d rather cry in a Jaguar than on a bus.”
― Françoise Sagan
Today let us see how we can implement a pivot table using java 8 streams. Raw data by itself does not deliver much insight to humans. We need some kind of data aggregation to discern patterns in raw data. A pivot table is one such instrument. Other more visual methods of aggregation include graphs and charts.
Sorting a large CSV file by loading it into SQLite. Much faster and easier to process.
“When you’re at the end of your rope, tie a knot and hold on.”
― Theodore Roosevelt
We are trying to sort a large CSV file. The file contains a couple of million rows – not large by “big-data” standards, but large enough to face problems working with it.
Large CSV files present a challenge when need arises to sort. Learn how to do that using a database.
“All of life is a constant education.”
― Eleanor Roosevelt, The Wisdom of Eleanor Roosevelt
Let us explore some ways of sorting large data sets.
By large, I don’t mean typical “big-data” sizes – which might consist of billions of rows. Such data sets fall into the realm of “big data” which we are not exploring today. Instead I am talking of sorting a rather large CSV file – maybe a couple of million rows.
Create an Excel Pivot table from Java using Apache POI.
“A foolish faith in authority is the worst enemy of truth.”
― Albert Einstein
A Pivot Table is a tool used in Excel for summarizing data. It helps group data using user-selected criteria and compute group summaries using functions such as total, average, count, etc.
Learn how to perform SQL-like grouping and summarizing calculations on Java Collections (List, Map, etc).
Have you wanted to perform SQL-like operations on data in a List or a Map? Maybe computing a sum or average? Or perhaps performing an aggregate operation such as summing a group? Well, with Java 8 streams operations, you are covered for some of these.
A previous article covered sums and averages on the whole data set. In this article, we show how to use Collectors.groupingBy() to perform SQL-like grouping on tabular data.
Convert CSV to JSON using Jackson. Use a POJO for conversion or a List & Map for intermediate storage.
“Any fool can know. The point is to understand.”
― Albert Einstein
CSV to JSON conversion is easy. In this article, we present a couple of methods to parse CSV data and convert it to JSON. The first method defines a POJO and uses simple string splitting to convert CSV data to POJO, which in turn is serialized to JSON. The second method uses a more complete CSV parser with support for quoted fields and commas embedded within fields. In this method, we use the Java Collection classes to store the parsed data and convert those to JSON.
More formatting options using Java with Apache POI for Microsoft Excel spreadsheets.
In Part 1 of this Apache POI Excel guide, we examined how to create an Excel spreadsheet and add data to it. We also looked at properly storing data into cells to avoid “Number Stored as Text” errors.
In this chapter, let us look at some more options for formatting data within an Excel spreadsheet.
Reading a CSV file in Java including handling BOM (Byte-Order-Marker), quoted fields, multi-line fields and more.
“The reason I talk to myself is because I’m the only one whose answers I accept.”
― George Carlin
CSV files are extensively used in data interchange between applications. Especially useful when the only structure to the data being exchanged is rows and columns. This format is particularly popular as the data can be imported into Microsoft Excel and used for charts and visualization.
