How to allocate an immense amount of memory?

Question

How to allocate an immense amount of memory?

Navigation

#1 by (6 votes)

7

I created an OnFly encryption algorithm that encrypts bytes without having to copy them to a new copy of these same bytes. This is both for encrypting and decrypting these byte lists. As in the scheme below, the same string that enters will be the same one that will be described, I did this thinking about the cost of memory I would have when using large files.

Forexample,considerthefileBytesfilethathasa4MBPNGimage(4,194,304bytes)storedinit.Myalgorithmonlyincreasesby1byteoftheoriginalsize,andencryptstheremainingbytesofthefile,maintainingitssizeandappearance.

Todecrypt,1byteisdiscountedinthisfile.

Myquestionisasfollows,ifIuseanarbitrarilylargefileinencryption,inadditiontousingahugeprocessorusagetheapplicationmemorywillbeaddedalongwiththefilesize,andifIusea10GBfileona4GBRAMcomputer,aOutOfMemoryExceptionexceptionwillbethrown.

Seeintheexamplebelowhowthiswouldoccur:

//askeysemCRYRAZtambémsãooutrasArraysdebytes.byte[]key=newbyte[]{12,51,63};//criaumanovainstânciadeCRYRAZ.CryrazclientCriptografia=newCryraz(key);//amensagemqueseráusadaparaencriptarstringMensagem="Olá, mundo!";
// bytes da mensagem
byte[] Mensagem_bytes = System.Text.Encoding.Default.GetBytes(Mensagem);

// Mensagem_bytes será o nosso alvo. Atualmente ela se encontra decriptada e original, vamos encriptar ela agora.
Cryraz.EncryptData(ref Mensagem_bytes);

// Pronto, Mensagem_bytes está encriptado.

This is how Cryraz works, unlike the other encryption methods that you copy the bytes, thus getting the original version and encrypted in the same code, different from Cryraz, it encrypts the original directly.

This is the body of the encryption method (I cut unnecessary parts of algorithm security checks and options):

public void EncryptData(ref byte[] entryByteData) {
    // ...
    for (int i = 0; i <= entryByteData.Length - 1; i++) {
        // ação recursiva em todos os bytes de entrada
        // é aqui a ação de criptografia
        // no final, o byte é atribuído à array:
        entryByteData[i] = ((byte)a);
        // obs: "a" é o novo byte encriptado
    }
    // aumenta o tamanho original da array para inserir o hash de segurança
    Array.Resize(ref entryByteData, entryByteData.Length + 1);

    // cria o hash de segurança e insere-o na array, sem esse hash, não há
    // descriptografia, é parte do algoritmo
    byte hash_x = performKeyHash(key);
    entryByteData[entryByteData.Length - 1] = hash_x;

    // coleta as variáveis inutilizadas
    GC.Collect();
}

Full Code: link

Even with GC.Collect(); at the end of the method there is a huge amount of memory being used in the application, because of reading the bytes of the files that will be used.

For the algorithm to work, it is necessary to know the byte index in relation to the array being encrypted and to have the exact size of the input and output.

In short, how do I get the application to "cache" all bytes of a very large file without any memory problems? Is there a way to "dribble" this?

c# .net criptografia

asked by anonymous 07.07.2017 / 20:53

1 answer

What is Service Container? What is WRAP and UNWRAP?

score 6 · Accepted Answer

Obviously you are prohibited from reading your entire file at one time. The only alternative you have is to process the file in blocks:

public static void Encripta(string src, string dest){
    Directory.CreateDirectory(dest);
    File.Create(dest);
    var buffer = new byte[4096];
    using(var reader = File.OpenRead(src))
    using(var writer = File.OpenWrite(dest))
    {
        int bytes;
        while((bytes = reader.Read(buffer, 0, buffer.Length)) > 0){
            //chama o seu encryptData com buffer aqui
            writer.Write(buffer, 0, bytes);
        }
    }
}

You will also have to adapt your current algorithm. Both the encryption part and the decryption part. From what I've always realized that it encrypts an array, the algorithm adds a byte at the end (a kind of checksum). But the problem now is that you just want this to be done in the last array.

public static void Encripta(string src, string dest){
    Directory.CreateDirectory(Path.GetDirectoryName(dest));
    var buffer = new byte[4096];
    var cipher = new Cryraz(new byte[]{10, 11, 12});

    using(var reader = File.OpenRead(src))
    using(var writer = File.Create(dest))
    {
        int bytes;
        while((bytes = reader.Read(buffer, 0, buffer.Length)) > 0){
            Array.Resize(ref buffer, bytes);
            //O algoritmo de encriptacao tem que ter um parametro adicional que indica se este é ou nao o último bloco.
            cipher.EncryptData(ref buffer, reader.Position == reader.Length);
            //chama o seu encryptData com buffer aqui
            writer.Write(buffer, 0, buffer.Length);
        }
        writer.Flush();
    }

}

Similarly, the decryption algorithm will always read the last byte of the array to get the checksum. The problem is that the checksum is only present in the last array. What this means is that you have to read the value beforehand.

public static void Desencripta(string src, string dest){
Directory.CreateDirectory(Path.GetDirectoryName(dest));
var buffer = new byte[4096];
var cipher = new Cryraz(new byte[]{10, 11, 12});

using(var reader = File.OpenRead(src))
using(var writer = File.Create(dest))
{
    reader.Position = reader.Length - 1;
    //le o checksum do fim do ficheiro, para decriptar todos os blocos
    var checkSum = reader.ReadByte();
    reader.Position  = 0;
    int bytes;
    while((bytes = reader.Read(buffer, 0, buffer.Length)) > 0){
        Array.Resize(ref buffer, bytes);
        cipher.DecryptData(ref buffer, (byte)checkSum);
        writer.Write(buffer, 0, buffer.Length);
    }
    writer.Flush();
}

Full Code