Synchronizing data on android

1

So, I'm in a project that needs to save a large mass of data coming from an api json in android's SQLite, something like 20,000 records, so you can work offline with no problems.

Only the synchronization of this mass of data is taking a long time in the application, about 10 to 15 min. And it is totally unfeasible for the user to wait all this time for downloading the data.

I also took a look at couchbase lite.

My question is, Does sqlite work well with a large amount of records? If yes, would it have a way to decrease the synchronization time of the data in order to use sqlite? How to save the entire json inside the couchbase.

Edited

Parsejson font and insert into bank

final DBController sett = new DBController(getBaseContext());
Calendar c = Calendar.getInstance();
SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");

final String formattedDate = df.format(c.getTime());

RequestQueue queue = Volley.newRequestQueue(this);
JsonObjectRequest jsonObjReq = new JsonObjectRequest(Request.Method.GET, url, null, new Response.Listener<JSONObject>() {
    @Override
    public void onResponse(JSONObject response) {
        try {
            JSONObject json = response.getJSONObject("DATA");
            Iterator keys = json.keys();

            while(keys.hasNext()) {
                String currentDynamicKey = (String)keys.next();
                if (!currentDynamicKey.equals("SYSTEM") && !currentDynamicKey.equals("SA3")) {
                    JSONObject currentDynamicValue = json.getJSONObject(currentDynamicKey);

                    JSONArray records = currentDynamicValue.getJSONArray("RECORD");
                    JSONArray dbo = (JSONArray) currentDynamicValue.getJSONObject("SETTINGS").get("DBO");
                    String createTable = dbo.getString(0).toString();

                    createTable = createTable.replace("\u0000", "");
                    createTable = createTable.replace("\u0000", "");

                    sett.dropTable(currentDynamicKey);
                    sett.createTable(createTable);

                    for(int i=0; i < records.length(); i++) {
                        JSONObject jsonOBject = records.getJSONObject(i);
                        Map<String, String> dataArray = new HashMap<String, String>(jsonOBject.length());

                        for (int x=0; x < jsonOBject.length(); x++) {
                            String key = jsonOBject.names().getString(x).toString();
                            String value = jsonOBject.getString(key).toString();

                            dataArray.put(key, value);
                        }

                        sett.insertDataHash(currentDynamicKey, dataArray);
                    }
                }
            }
        } catch (JSONException e) {
            e.printStackTrace();
            Log.e("---[Erro no sync]---", e.getMessage());
        }
    }
}, new Response.ErrorListener() {
    @Override
    public void onErrorResponse(VolleyError error) {
        Log.d("---[Volley]---", "Error: " + error.getMessage());
        hidepDialog();
    }
}){
    @Override
    public Map<String, String> getHeaders() throws AuthFailureError {
        HashMap<String, String> headers = new HashMap<String, String>();
        headers.put("Accept", "application/json");
        headers.put("Content-type", "application/json");
        headers.put("Token", appToken);
        headers.put("DateTime", formattedDate);
        headers.put("Mac", deviceMac);
        headers.put("user", user);
        return headers;
    }
};
queue.add(jsonObjReq);

Json

{
  "RET": true,
  "DATA": {
    "SYSTEM": {
      "SYNC": true
    },
    "TABELA1": {
      "RECORD": [
        {
          "CAMPO1": "AAAAA",
          "CAMPO2": "00000",
          "CAMPO3": "AAA00"
        },
        {
          "CAMPO1": "BBBBBB",
          "CAMPO2": "111111",
          "CAMPO3": "BBB111"
        },
      ]
    },
    "TABELA2": {
      "RECORD": [
        {
          "CAMPO1": "CCCCCC",
          "CAMPO2": "000000",
          "CAMPO3": "CCCC00"
        },
        {
          "CAMPO1": "DDDDDD",
          "CAMPO2": "222222",
          "CAMPO3": "DDD222"
        },
      ]
    }    
  }
}
    
asked by anonymous 11.10.2016 / 17:07

2 answers

2

I was able to solve the problem of slowness by changing the api, now instead of bringing the fields and assembling the insert into the device, I already bring the api insert ready to execute in the device. Sync dropped for 30 sec.

    
28.10.2016 / 02:31
2

I currently work on a project where we have two tables with nothing less than 200mil records in one and 2.5 million in another. That's right, millions. In my case, the user does an "initial charge" before leaving for fieldwork. This load is done via Wi-Fi as follows: I query the oracle database, save the result in csv files, then I use the "import" command from sqlite to import csv into the sqlite database ( link ). After importing the data, I upload the database to Android (normal process of serialization and upload). Already in Android the performance even with this mass of data is not harmed in any way.

    
12.10.2016 / 02:51