Monday, September 24, 2012

Webcam Swiper


I've spent a little time lately working on an experiment that uses the new getUserMedia() method.  I'm calling it the Webcam Swiper.  Chrome and Firefox now support Webcam access, although Firefox < 20 requires turning on a flag in about:config.

It's a page that displays a book.  If you allow webcam access it will watch for you swiping your hand left or right through the air, which triggers a page turn in the appropriate direction.

I was inspired to try this when I saw the cool Magic Xylophone.  I thought of all the cases where someone might want to make a simple directional gesture and have something happen on the page.  Image Carousels could rotate, the next item in a list could be displayed, book pages could be turned, etc...

To implement this I had to figure out how to detect when a hand had moved from one side of the screen to the other. I started by detecting motion the same way that the Magic Xylophone does. First turn the image from the webcam to greyscale so I only have to worry about brightness and not color changes. Then, comparing one frame to the next, watch for any pixels that change intensity more than some given threshold.  Next I determine what I call the "motion weight" for the given frame. Basically any pixel on the left side of the screen which is considered to have movement, subtracts from the weight. Any pixel on the right side of the screen which has movement adds to the weight. Total them all up and we get a number that summarizes where the most movement is occurring on the screen. Now all we need to do is compare the "motion weight" over the last series of frames. If we have a highly negative weight which then quickly shifts to a highly positive weight, we know the user just swiped right. If it is positive and goes negative, the user just swiped left. A custom event is fired on the body element of the page. Then whoever is listening for the event can do whatever they want.  In my case I do a couple CSS transitions and transforms to turn the page of the book.

That's all there is to it. I had some problems with varying light level, so I sample every 50 frames and calibrate the sensitivity based on the ambient light level in the room. Also I adjust for slower hardware by timing each frame analysis. If it is taking too long I lower the scan frequency. Otherwise it stays high.

I've made this all into a library which should be fairly easy to use.

Include the webcam-swiper-0.1.js with a script tag or the loader of your choice.

Then bind the swipe events however you choose.  Example with jQuery:


  $("body").bind("webcamSwipeLeft", yourLeftEventHandler);
  $("body").bind("webcamSwipeRight", yourRightEventHandler);

Start the webcam access with a call to the global initializeWebcamSwiper function like this:


  window.initializeWebcamSwiper();

Now it is running!  If you choose to stop it call

  window.destroyWebcamSwiper();

Play with it and feel free to improve it or give any feedback.

25 comments:

  1. Are you kidding me? This is the coolest thing I've seen in a while! Downloading it now to mess with it. Thank you for sharing.

    ReplyDelete
    Replies
    1. Thanks! Glad you got something out of it.

      Delete
  2. is it possible to detect micro coordinate based motion on top of left or right movement? I.e can you paint a picture or sign your name in the air with a finger brush? @chamilyan

    ReplyDelete
    Replies
    1. It's difficult to identify small points... Something I have considered is starting a project that used fiducials (http://www.youtube.com/watch?v=eWwp19ymChY). That has the most promise when using relatively low fidelity webcams.

      Delete
    2. How about up/down or width between 2 points. (If you hold your hands apart). That wouldn't require too much precision? Those c++ projects are interesting too if you can also interpolate it into JS.

      Delete
  3. Great work. Could you make a demo video of how this works? Do you watch H+? Those gestures are all over the place. http://www.youtube.com/watch?v=MVekrZ-H5Pc

    ReplyDelete
    Replies
    1. I'd never seen that before (H+). Interesting. I'll see about making a demo video. Thanks for the feedback.

      Delete
  4. Awesome. I worked on a similar project in C++ two years ago : http://labs.steren.fr/category/no-touch-screen/

    I'm glad to see that today, you can achieve the same result in a browser. Congratulations for the work.

    ReplyDelete
    Replies
    1. Wow! That is similar. Looks like a similar technique. Good job on yours too!

      Delete
  5. This is neat. I have a disability so swiping hands isn't too easy... but I managed some success swiping my head sideways!

    If there was some way to feedback where it's looking (what the cam sees) and a method of defining a swipe area it'd be useful.

    Good stuff.

    ReplyDelete
  6. It would be nice to combine both swipe and touch for a "menu".

    ReplyDelete
  7. Dear Brandon!
    first of the all i would like to admire the effort. thats really owesum.

    but can you please guide me about the edit.
    how can i add the pictures instead of book?
    waiting for your reply.
    Thanks

    ReplyDelete
    Replies
    1. I would recommend making the page do what you want (such as rotate through images) based on click events... and once you have that working switch over to using the webcamSwipeEvents, which would be trivial.

      Delete
  8. How do you turn your webcam back off??

    ReplyDelete
    Replies
    1. Yeah, leaving the page or closing the tab will turn off the camera.

      Delete
    2. it wasn't turning off when I left the page or closed the tab ;)

      Delete
  9. It works when is open other tab, this is bad.

    ReplyDelete
  10. Real Awesome . I have been looking up the whole getUserMedia for a while. This is a really good library. Awesome work

    ReplyDelete
  11. a other way in css. i think you can now kill your little problem in the loop handling. best regards - i like it

    http://www.devoteem.de/_files/bookdemo_package_jquery.zip

    ReplyDelete
  12. Awesome, man!! Coolest thing I have seen over the last few weeks!!

    ReplyDelete
  13. And here is WebcamSlapper! A demo, that uses WebcamSwiper to control HTML5 Video: http://creativecollective.gr/labs/webslap/

    ReplyDelete