View unanswered posts | View active topics It is currently Sun Jan 19, 2020 9:42 am

Reply to topic  [ 2 posts ] 
 Experiment with Vision Sensing Using fitlet2 and Pixy2 
Author Message

Joined: Mon May 28, 2012 12:25 pm
Posts: 106
Post Experiment with Vision Sensing Using fitlet2 and Pixy2

Vision Sensing.jpg
Vision Sensing.jpg [ 8.84 KiB | Viewed 2207 times ]

Vision Sensing represents by far the best overall method of perceiving the surrounding environment. While other sensory technologies such as millimeter wave radar, lidar, acoustic, thermal, chemical , and so on are very useful, none of them provides the level, quality, and variety of information that vision does.

Previously, electro-optical sensory systems were quite expensive and were mostly found in military hardware such as the AGM-65 Maverick missile and similar systems deployed around the world. Now however, affordable and capable vision sensing devices are available to the rest of us. One such device is the Pixy2 camera which is capable of line following and object recognition. Available for around $60 US, Pixy2 is a great addition to small, capable platforms such as fitlet2.

Older Machine Vision Sensing Technology - Expensive and Limited to Military Applications
Maverick.jpg [ 30.48 KiB | Viewed 2207 times ]

Pixy2 - Modern Machine Vision Sensing Technology - Affordable and Available to All
Pixy2.jpg [ 46.29 KiB | Viewed 2207 times ]

About Pixy2

Pixy2 is the latest incarnation of Pixy which was born of a partnership between the Carnegie Mellon Robotics Institute and Charmed Labs. The hardware and software utilized in Pixy2 are open source, so it is an ideal device for use by the makers among us.

Pixy2 Open Source.jpg
Pixy2 Open Source.jpg [ 8.58 KiB | Viewed 2204 times ]

Pixy2 has a couple of sensing modes that are quite useful for the maker - especially for robotics applications. These modes are:

1. Object Detection and Recognition

Pixy2 learns and detects objects based on colors using an algorithm known as Color Connected Components. Color based object recognition was utilized because of its efficiency, speed, and relatively "bullet-proof" performance. Pixy2 has the ability to simultaneously detect and track a large number of objects simultaneously. This is a capability that was once reserved for only very complex and expensive military systems.

Orange.png [ 126.91 KiB | Viewed 2207 times ]

2. Line Tracking

Pixy2 has the ability to track and follow lines using its custom line tracking algorithm. Unlike photodetector based solutions, Pixy2's vision based line tracking solution has the ability to anticipate by looking ahead and the ability to easily identify and navigate through intersections.

Line Tracking 1.png
Line Tracking 1.png [ 42.06 KiB | Viewed 2207 times ]

Hardware Used

Hardware.jpg [ 26.33 KiB | Viewed 2207 times ]

The hardware used for this demonstration is:

1. fitlet2
2. Pixy2 camera system
3. Pixy2 Pan-Tilt kit


The basic software package for configuring and experimenting with Pixy2 is PixyMon. It is available for Linux, Mac, and Windows. This app note focuses on the Linux version. The preferred distro for the Linux version of PixyMon is Linux Mint which is what I use.

LM Friendly.jpg
LM Friendly.jpg [ 6.85 KiB | Viewed 2204 times ]

Software Installation

Detailed instructions for installing PixyMon in Linux are found here: ... n_on_linux

In my case I installed PixyMon in /usr/local/src/pixy2 in order to adhere to what I perceive to be standard Linux practice - although there are many different opinions concerning what is standard practice in Linux.

Also, I wrote a short script which I made executable and placed in /home/[username]/bin for universal access.
# Script to Run PixyMon
cd /

Test Results

Pixy2 in conjunction with fitlet2 proved to be a fun and interesting system to explore. I was impressed with Pixy2's ability to accurately and consistently detect, identify, and track objects. The following images and videos provide more detail.

1. Object Detection, Recognition, and Tracking

Basic Shape/Color Recognition - Primary Colors
Primary.jpg [ 143.77 KiB | Viewed 2204 times ]

Basic Shape/Color Recognition - Blended Secondary Colors
Secondary.jpg [ 95.87 KiB | Viewed 2204 times ]

Basic Shape/Color Recognition - Multiple Objects
Multi.jpeg [ 150.4 KiB | Viewed 2204 times ]

Basic Shape/Color Tracking - PixyMon - Ripe Tomato

Basic Shape/Color Tracking - Pixy2 Pan/Tilt Kit - Ripe Tomato

2. Line Tracking

Since I am using Pixy2 in a more or less static configuration, I was not able to make a detailed exploration of its line tracking performance. However, the following picture shows that Pixy2 does its best to search its field of view for lines to follow.

Line Tracking.png
Line Tracking.png [ 65.52 KiB | Viewed 2204 times ]

Potential Projects

1. Lunar Tracker for aiming "Moon Bounce" laser and microwave transmissions.
2. Laser tracker for mobile platform.
3. Automated fruit/vegetable harvesting system using color/shape recognition.
4. Vehicle "road ahead" hazard detection, recognition, and warning system.
5. Etc.

Actual Line Following Project (By Others)

Personally, I think a fitlet2 would make a fine field-hardened replacement for the laptop used in the following video.

Resources for Developers, Experimentors, and Makers

APIs, Libraries, and Software



Where Do fitlet2 and Pixy2 Go From Here?

Where To.jpeg
Where To.jpeg [ 4.17 KiB | Viewed 2204 times ]

That remains to be seen, but I bet they could be the foundation for a Very Cool Robot!!

Mon Aug 26, 2019 12:26 am
Site Admin

Joined: Mon Dec 25, 2017 4:21 pm
Posts: 394
Post Re: Experiment with Vision Sensing Using fitlet2 and Pixy2
Very interesting stuff right there.
Thank you for this enriching post.

Mon Aug 26, 2019 8:29 am
Display posts from previous:  Sort by  
Reply to topic   [ 2 posts ] 

Who is online

Users browsing this forum: No registered users and 4 guests

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by STSoftware for PTF.