Home / Tutorials / Arduino Tutorial / Detect Objects with Camera and Arduino

Detect Objects with Camera and Arduino

The basic Arduino boards are not powerful enough for image processing. Heck, they are not even good enough to capture photos without the help of a desktop computer. However, there is a special imaging device you can use to make an Arduino detect objects.

The special device I am referring to is the Pixy camera. This camera incorporates a microprocessor which does all the heavy image processing stuff and simplifies object detection. This tutorial shows you how to use the Pixy camera to build an Arduino object detection project.

Setting Up Object Signatures

The Pixy camera board comes with two cables: one to be connected to an Arduino and another to be connected to a desktop or laptop. See image below:

Arduino UNO and Pixy connection

As you can see, the Pixy directly supports Arduino boards with ISP headers such as an Arduino UNO, Nano and Mega. Other boards may be used as well since the Pixy board uses SPI communication. Just take note of the following pin definitions:

Pixy camera pinout
From docs.pixycam.com

Next, download Pixymon, a desktop app that allows you to set which objects the camera will be able to recognize.

After downloading and installing Pixymon, start it and you should be able to see a stream from the camera.

Pixymon video stream

Here’s the catch: the Pixy camera recognizes objects through their color. You can save up to seven different color which are called “signatures”. This means that two objects with the same color will be recognized as one.

To set up a signature, press and hold the button on the board. The LED on the board should start changing color. Release the button once the LED color changes to red. Pixymon will now engage signature setup mode and will ask you to place an object in front of the camera. If a color is distinct enough, it should appear to be shaded on the Pixymon screen.

Pixymon color signature
Press the on-board button again to save this color signature.

The Pixy camera should now be able to detect and track the object, wrapping the object in a rectangle with the text “s=1”.

Pixymon object detected

You can add more signatures by repeating the process above except waiting for the red color on the LED. To setup a second color signature, release the button on the color immediately following red. Similarly, setup a third color signature by releasing the button on the color immediately following the color after red, and so on.

To learn more about using Pixymon, read its official wiki page.

Pixy and Arduino

First, download the Pixy library for Arduino. Then, open one of the examples provided on the
said library, ccc_hello_world.

Upload this sketch and then open serial monitor with baud rate equal to 115200.

If the Pixy detects an object, the following text should appear on the serial monitor:

Arduino Pixy serial output

There are several information found here:

block is the number of objects detected,
x and y are the coordinates of the object in the frame,
sig is the color signature,
width and height are the dimensions of the rectangle which envelopes the object.
index is the tracking index, which is a number assigned to a detected object. This number is useful for tracking certain objects with the same color signature in a single frame.
● Finally, age is the the number of frames the camera has captured since starting.

Detecting a Specific Object

We can use the data on the serial monitor to determine specific objects. For example, we want to detect a pencil.

First, setup the pencil as a color signature by following the setup above. A common pencil would have different colors for its body and eraser head. Thus, we will create two color signatures.

Arduino Pixy detecting pencil

Next, we create a sketch based on the ccc_hello_world example above. We need to check the width and height of the detected signatures and determine which size suggests a pencil. With this technique, we can reduce the detectable objects to those with the same colors and size of a pencil.

My serial monitor shows this when the pencil is captured inside the frame:

Signature 1 is the pencil body and signature 2 is the eraser head. According to the information above, signature 1 has a size of 254 x 30 while signature 2 has a size of 14 x 4. The actual width and height are irrelevant here; what we need is the difference between the sizes of each signature. With this, we can say that there is an object with a large rectangular yellow body and another object with a small rectangular red body.

We can also use the location of each object by looking at the x and y’s. If the image of the pencil is horizontal, the body and the eraser head should be at about the same level vertically. Here, both signatures lie within 100 to 120 range.

Here’s now a sketch that detects a pencil using the Pixy camera:

Here, I used the area of the pencil’s body and eraser head and compare them with each other. Based on my initial tests, the pencil’s body is about 100 times the size of the eraser head, hence this statement:

Next, the location of the pencil’s body and eraser head should be about the same level in x if the pencil is horizontal or in y if the pencil is vertical. Hence,

Only when both area and location is satisfied will we print that a pencil has been detected

That’s it! I will be building a lot of projects with the Pixy camera so I hope you come back!

Check Also

Sensor attached to Arduino UNO

Using a Fingerprint Sensor

Fingerprint sensors are old technology but it wasn’t readily available to makers until it was …


  1. Can you help us on how to code pixy and arduino on avoiding objects?

    • in my opinion, you should make it check for certain objects and if it detects one of the objects it wants to avoid you make it walk away from it. this could be done by making a function that is just meant to make the robot turn around and calling it whenever the desired object is detected.

Leave a Reply

Your email address will not be published. Required fields are marked *