What better way to parse a very large json file in java?

As my real need in this Json was for some tags, what I used was an element-by-element reading at each iteration, according to the need to use it. For this I used the jackson Json API. My code below takes only the title, url, text, entidades tags from the above Json:

public class BrutoNewsJsonParser {

    JsonFactory factory;
    JsonParser jp;
    JsonToken current;

    public BrutoNewsJsonParser() {
        factory = new JsonFactory();
        jp = null;

        String path = "/home/nicolas/Documentos/X9dadosIC/Bruto/news_jul_dez_2013.json";

        try {
            jp = factory.createJsonParser(new File(path));

        } catch (IOException ex) {
            Logger.getLogger(BrutoNewsJsonParser.class.getName()).log(Level.SEVERE, null, ex);
            ex.printStackTrace();
        }

    }

    public News ler() {
        EntidadesReader er = new EntidadesReader();
        String title = null, url = null, text = null;
        LinkedList<String> entidades = new LinkedList<>();
        boolean controleEntidades = true;

        int contador = 0;

        try {
            current = jp.nextToken();
        } catch (IOException ex) {
            Logger.getLogger(BrutoNewsJsonParser.class.getName()).log(Level.SEVERE, null, ex);
        }

        if (current == JsonToken.START_OBJECT) {
            contador++;
        }

        while (contador != 0) {
            try {
                String namefield = jp.getCurrentName();
                if ("title".equals(namefield)) {
                    title = jp.getText();
                } else if ("url".equals(namefield)) {
                    url = jp.getText();
                } else if ("text".equals(namefield)) {
                    text = jp.getText();
                } else if ("entities".equals(namefield) && controleEntidades) {
                    if (current == JsonToken.START_ARRAY) {
                        controleEntidades = false;
                        current = jp.nextToken();
                        while (current != JsonToken.END_ARRAY) {
                            entidades.add(er.traduzir(Integer.parseInt(jp.getText())));
                            current = jp.nextToken();
                        }
                    }
                }

                current = jp.nextToken();
                if (current == JsonToken.END_OBJECT) {
                    contador--;
                } else if (current == JsonToken.START_OBJECT) {
                    contador++;
                }
            } catch (IOException e) {
                System.err.println(current.asString());
                e.printStackTrace();
            }
        }
        try {
            jp.nextToken();
        } catch (JsonParseException j) {

        } catch (IOException ex) {
            Logger.getLogger(BrutoNewsJsonParser.class.getName()).log(Level.SEVERE, null, ex);
        }
        return new News(title, url, text, entidades);
    }
}

With this, each Read method call provides an additional element.

The database is probably the best solution because:

• It is made to work with huge amounts of data;

• 5GB is very much meant to be kept in memory, especially in Java;

• If the data needs to be reused, it will be necessary to perform the entire process of interpretation, etc. of the data, which will undoubtedly take time.

I do not know of a specific tool that manages to handle so much data like that, or that it boasts of being able to manipulate so much data. But unless you have more than 4 ^ 31 records in a single level of your object tree, enough memory on your machine and set Java to have a really large (8GB +) heap limit, I see no problems. / p>

A detail to note that can make it much easier: if your file consists only of lines like the one described, and nothing else, if you want separated by commas, and each line is a complete Json, you can process row by line as being a Json file, and then sending the line to the database, eliminating the previously cited memory problem, with the advantage of being reasonably simple to do.