I need a crawler that does pagination on a website.
I'm reading the source code and generating a txt in this way
public class CodFonte {
public static void crawler(String str) throws IOException {
URL url = new URL(str);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setReadTimeout(15 * 1000);
connection.connect();
// read the output from the server
BufferedReader reader = new BufferedReader(new InputStreamReader(
connection.getInputStream()));
StringBuilder stringBuilder = new StringBuilder();
String linha = "";
String path = System.getProperty("user.home") + "\Desktop\";
String fileName = "Fonte Code.txt"; // Nome do arquivo
FileWriter file = new FileWriter(path + fileName);
PrintWriter gravarArq = new PrintWriter(file);
gravarArq.println("SITE -------- " + url);
while ((linha = reader.readLine()) != null) {
gravarArq.println(linha);
}
file.close();
reader.close();
}
}
But I need to go to the next page, the url is friendly does not change according to the form request that is via POST.