How to manipulate the FPU stack in the x86 assembly using the NASM and functions in c?

The logic in using the FPU is correct.

There are only two problems in the program:

1) Data upload to FPU

When executing the scanf function to read the data, the formatted string entered is %d , so the read result will store an integer in the destination memory.

When you load this number into the FPU, such as in the fld dword[red] line, the FPU expects to find a number in the form Point Floating , not an integer.

The solution is to change the format string scan to "%f" .

2) Storing the calculated result in memory

The FPU is trying to store a result in qword format, however, you specified the destination address as dword .

As the value to be stored is larger than the destination, the FPU throws an exception of type #P , which according to the Intel manual:

  

"Value can not be represented exactly in destination format."

The solution is to change the prefix dword to qword .

Below is the program already with these fixes:

global _main
extern _printf  ;a função _printf em C é chamada
extern _scanf   ;a função _scanf em C é chamada

SECTION .data

msg1        db  "Pixel vai ficar preto!", 10,0
msg2        db  "Pixel vai ficar branco!", 10,0
msg3        db  "Digite os valores para as cores vermelho, verde e azul:", 10,0
scan        db  "%f", 0 ; tem que ser %f!!!!

;msg4 db "Os numeros sao %d, %d, %d", 0
msg5        db "valor final %.3f",10,0
constante1  dd 0.299
constante2  dd 0.587
constante3  dd 0.114
constante4  dd 0.713
constante5  dd 128
constante6  dd 0.564
;y dw 0

SECTION .bss

red         resd 1
green       resd 1
blue        resd 1
y           resd 1
cr          resd 1
cb          resd 1

SECTION .text

_main:
            ; Aqui, precisa salvar o stack frame
            push        ebp
            mov         ebp, esp

            push        msg3
            call        _printf
            add         esp, 4

            ;pega o valor de vermelho
            push        red
            push        scan
            call        _scanf
            add         esp, 8

            ;pega o valor de verde
            push        green
            push        scan
            call        _scanf
            add         esp, 8

            ;pega o valor de azul
            push        blue
            push        scan
            call        _scanf
            add         esp, 8


;imprime os valores lidos (só pra teste)
;push dword[blue]
;push dword[green]
;push dword[red]
;push msg4
;call _printf
;add esp, 16

calcula_y:

            fld         dword[red]          ;coloca na pilha o valor de red
            fld         dword[constante1]   ;coloca na pilha o valor da constate1
            fmulp       st1, st0            ;faz a multiplicação e salva em st1

            fld         dword[green]        ;coloca na pilha o valor de green
            fld         dword[constante2]   ;coloca na pilha o valor da constate2
            fmulp       st1, st0            ;faz a multiplicação e salva em st1

            fld         dword[blue]         ;coloca na pilha o valor de blue
            fld         dword[constante3]   ;coloca na pilha o valor da constate3
            fmulp       st1, st0            ;faz a multiplicação e salva em st1

            faddp       st1, st0            ;faz a adição de st1 e st0 e salva em st1 e dá um pop em st0
            faddp       st1                 ;faz a adição de st1 e st0 e salva em st0
            fst         qword[y]            ; Aqui, tem que ser QWORD

            push        dword[y+4]
            push        dword[y]
            push        msg5
            call        _printf
            ;call        _scanf
            add         esp, 12

            mov         esp, ebp
            pop         ebp
            ret

The program was compiled with NASM version 2.11.08 , using the command:

nasm -g -f win32 xx.asm

And linked with Microsoft (R) Incremental Linker Version 14.00.23506.0 , through the command:

link /debug:FULL /libpath:"caminho_para_o_dir_lib_do_msvc" /subsystem:CONSOLE xx.obj "libcmt.lib" "advapi32.lib" "legacy_stdio_definitions.lib"

Change path_to_dir_lib_of_msvc to the lib directory within the MSVC installation.

For the debugging process during the analysis, the program was used: x64dbg (32-bit version).

After running, the result of the program was:

Digite os valores para as cores vermelho, verde e azul:
100.0
101.0
102.0
valor final 100.815

Note: To perform the link process without errors, it was necessary to add the underline '_' before the functions.

For more references, see the manuals (in PDF):
Intel® 64 and IA-32 Architectures Software Developer Manuals