Help image processing (background removal)

2

I need to create a code to perform the frames processing of a video. In case the program will scan the pixels of the image and analyze the colors of the pixels and if you find colors close to the color initially defined, you should replace it, for example, with a green chroma key.

I would like tips on this: what library to use, how can I make the color check be next to the color that I will define (I was told to use the RGB band, but I was not told very specifically what to do). >

If they have some sample code it would also be very useful.

EDIT:

What I do not really know is how to treat the image.

My goal is to read all the pixels on the screen and check their color. In this case, I want to check if the pixels are similar to the orange shade I'll choose, to be more specific. If the program finds a certain pixel that is similar to this orange (lighter, darker, etc ...), you should replace it with a green chroma key.

I do not know how to perform this pixel comparison. I was advised to use the RGB band of each pixel to determine its similarity to orange (through a "threshold" of similarity). But I do not know if this would be the best way to compare the pixels or how to compare the RGB values to see if the pixel found is similar to orange (it's within the threshold if similarity).

    
asked by anonymous 29.12.2015 / 03:48

1 answer

6

As I've already commented, this other question has a similar problem (but in C #). The trivial solution is to use the thresholding process: you scan all the pixels in the image and change to the desired color (green) only those pixels that are below or above a chosen threshold.

I made a sample program similar to the one that is in my answer of that other question (that is, that C # code for Java):

import javax.swing.JFrame;
import javax.swing.JPanel;
import javax.swing.JButton;
import javax.swing.JFileChooser;
import javax.imageio.ImageIO;
import javax.swing.BorderFactory;
import javax.swing.Box;
import javax.swing.BoxLayout;
import javax.swing.ImageIcon;
import javax.swing.JLabel;
import javax.swing.JOptionPane;

import java.awt.BorderLayout;
import java.awt.Color;
import java.awt.Component;
import java.awt.Dimension;
import java.awt.Graphics2D;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;

@SuppressWarnings("serial")
public class Limiar extends JFrame implements ActionListener {

    private JLabel m_oImage;

    private JButton m_oLoadButton;
    private JButton m_oDetectButton;

    private JPanel m_oButtonsPanel;

    final JFileChooser m_oFileChooser = new JFileChooser();

    public Limiar() {
        super("Exemplo de Limiarização");

        m_oImage = new JLabel();
        m_oImage.setHorizontalAlignment(JLabel.CENTER);
        m_oImage.setVerticalAlignment(JLabel.CENTER);
        add(m_oImage, BorderLayout.CENTER);

        m_oButtonsPanel = new JPanel();
        m_oButtonsPanel.setLayout(new BoxLayout(m_oButtonsPanel, BoxLayout.Y_AXIS));
        m_oButtonsPanel.setBorder(BorderFactory.createEmptyBorder(10,10,10,10));
        add(m_oButtonsPanel, BorderLayout.EAST);

        m_oLoadButton = new JButton("Carregar Imagem...");
        m_oLoadButton.addActionListener(this);
        m_oLoadButton.setActionCommand("Load");
        m_oLoadButton.setAlignmentX(Component.CENTER_ALIGNMENT);
        m_oButtonsPanel.add(m_oLoadButton);

        m_oButtonsPanel.add(Box.createRigidArea(new Dimension(0, 10)));

        m_oDetectButton = new JButton("Substituir fundo");
        m_oDetectButton.addActionListener(this);
        m_oDetectButton.setActionCommand("Detect");
        m_oDetectButton.setAlignmentX(Component.CENTER_ALIGNMENT);
        m_oButtonsPanel.add(m_oDetectButton);
    }

    @Override
    public void actionPerformed(ActionEvent oEvent) {
        if (oEvent.getActionCommand().equals("Load")) {

            int iRet = m_oFileChooser.showOpenDialog(this);

            if (iRet == JFileChooser.APPROVE_OPTION) {
                File oFile = m_oFileChooser.getSelectedFile();

                BufferedImage oImg;
                ImageIcon oIcon;
                try {
                    oImg = ImageIO.read(new File(oFile.getPath()));
                    oIcon = new ImageIcon(oImg);
                    m_oImage.setIcon(oIcon);
                } catch (IOException e) {
                    JOptionPane.showMessageDialog(null, "Não foi possível abrir a imagem selecionada");
                }

            }
        }
        else if (oEvent.getActionCommand().equals("Detect")) {

            final double THRESHOLD = 127.5;

            ImageIcon oIcon = (ImageIcon) m_oImage.getIcon();
            BufferedImage oImg = new BufferedImage(oIcon.getIconWidth(), oIcon.getIconHeight(), BufferedImage.TYPE_INT_ARGB);

            Graphics2D oGr = oImg.createGraphics();
            oIcon.paintIcon(null, oGr, 0, 0);
            oGr.dispose();

            double dBrightness;
            Color oColor;

            for (int x = 0; x < oImg.getWidth(); x++) {
                for (int y = 0; y < oImg.getHeight(); y++) {
                    oColor = new Color(oImg.getRGB(x, y));

                    dBrightness = 0.21 * oColor.getRed() + 0.72 * oColor.getGreen()  + 0.07 * oColor.getBlue();
                    if (dBrightness > THRESHOLD) {
                        oImg.setRGB(x, y, Color.GREEN.getRGB());
                    }
                }
            }

            m_oImage.setIcon(new ImageIcon(oImg));
        }
    }   

    public static void main(String[] args) {
        Limiar t = new Limiar();

        t.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        t.setSize(600,480);
        t.setLocationRelativeTo(null);
        t.setVisible(true);
    }   
}

The result (using the same tattoo image of the question quoted) is as follows (the background is changed to green, and the rest is kept unchanged):

Screen after loading the tattoo image:

Screenafterprocessingthebackgroundreplacement:

Importantdetails

  • ThereintheoriginalcodeinC#therewasalreadyamethod(getBrightness)toreturnthebrightness(ieconvertcolorto3bandsforgrayscale).IdonotknowifthereisamethodforthisinJava(atleastIdidnotfindanythingintheColorclass).SoI'veconvertedmyself(withonethat'sexplained in my other answer ).

  • The values of a pixel (either in RGB or grayscale) are normalized in C # as float between 0.0 and 1.0 . In the case of Java, they are values between 0.0 and 255.0 . So the initial threshold chosen in that original code was 0.5 and here it is 127.5 .

  • The case of the other question was simpler because the images were more standardized with a lighter background than the rest (they always had a white background). You do not give details about the video you are processing (not even an example! Tsc, tsc, tsc), but I imagine the background will not have a very simple pattern. You will surely need to change the threshold value to do some testing. And if the background is darker than the rest of the image, you'll also need to change the brightness comparison operator from > to < (to detect and change only the darker - remember how much the closer the 0 , the darker the pixel is). For example, the processing of this famous meme follows:

  • Thiswillworklikethiswiththecurrentcode:

    Andso,ifyoujustswitchtheoperatorto<:

    Becauseinthisimagethebackgroundisdarkerthanthecat,theexchangeworksbetter.However,notehowthealgorithmfailstoincludethecat'stieandglassesaspartofthe"background." In cases like these, you could make far more detailed comparisons, such as checking whether the pixel is in a range of brightness, or whether it is included in a large connected region (that is, whether it is most likely a background or not).

  • Alternatively to the thresholding method, it may be possible to make statistical segmentations by analyzing the probabilities of the color histogram (something similar to what I describe in this my other answer ) or with the training of a decision tree. But this will depend on your problem domain (and just to name one more time, you unfortunately have not provided any information about it, despite the many requests).

  • Keep in mind that rendering targeting for each frame in a video will be computationally costly and may slow render output. You may have to work to improve performance or rethink the problem.

  • 10.01.2016 / 20:53