Anaglyphs and 3D vision

Oh a post about vision systems (about time!)

So you can write alot of code to try and interpret the 3D space in images or you can cheat and use the same technique that the brian uses (well it probably does not use pointers but thats another conversation)

So if you have two cameras you can position them side by side like eyes, and using a bit of DirectX coding take some pictures at the same time:

In the pictures above you can see that the images are slightly offset from each other (the wall on the left is not visible in the right picture), and from this you can generate Anaglyph image:

Now if you put your 3D spec's on (the kind that came with Shrek 3D DVD) you can see it in 3D!

The code? something like this does the trick (cue screams of fear! its unsafe code! it's got pointers and stuff! hide behind the sofa!

unsafe public Bitmap MakeAnaglyph()
  {
   Bitmap result = (Bitmap)_right.Clone();

   FastBitmap fbitRight = new FastBitmap(result, ImageLockMode.ReadWrite);
   FastBitmap fbitLeft = new FastBitmap(_left);

   FastBitmap.PixelData* pPixelRight;
   FastBitmap.PixelData* pPixelLeft;

   for (int y = 0; y < _right.Height; y++)
   {
    for (int x = 0; x < _right.Width; x++)
    {
     pPixelRight = fbitRight[x, y];
     pPixelLeft = fbitLeft[x, y];

     pPixelRight->red = pPixelLeft->red;
     pPixelRight->green = (byte)(pPixelRight->green * .82);
     pPixelRight->blue = (byte)(pPixelRight->blue * .95);
    }
   }

   fbitRight.Dispose();
   fbitLeft.Dispose();

   return result;
  }

This code uses my FastBitmap class, which justs makes working with the raw data a bit more obvious:

unsafe class FastBitmap: IDisposable
 {
  Bitmap _source;
  Rectangle _bounds;
  BitmapData _lockedBitmap;
  byte* _pBase;
  int _stride;

  [StructLayout(LayoutKind.Sequential)]
  public struct PixelData
  {
   public byte blue;
   public byte green;
   public byte red;
  }

  public FastBitmap(Bitmap source): this(source, ImageLockMode.ReadOnly)
  {
  }

  public FastBitmap(Bitmap source, ImageLockMode mode)
  {
   _source = source;
   _bounds = Bounds;
   _lockedBitmap = _source.LockBits(_bounds, mode, source.PixelFormat);
   _pBase = (byte*)(void*)_lockedBitmap.Scan0;
   _stride = _lockedBitmap.Stride;
  }
  public PixelData* this[int x, int y]
  {
   get
   {
    // Format32bppArgb - Specifies that the format is 32 bits per pixel; 
    // 8 bits each are used for the alpha, red, green, and blue components
    const int bytesPerPixel = 4; 

    PixelData* result = (PixelData*)(_pBase + (y * _stride) + (x * bytesPerPixel));

    return result;
   }
  }

  public Rectangle Bounds
  {
   get
   {
    GraphicsUnit unit = GraphicsUnit.Pixel;
    RectangleF bounds = _source.GetBounds(ref unit);

    Rectangle result = SizeToRectangle(bounds.Location, bounds.Size.ToSize());

    return result;
   }
  }


  private Rectangle SizeToRectangle(Size size)
  {
   return SizeToRectangle(new Point(0, 0), size);
  }

  private Rectangle SizeToRectangle(Point point, Size size)
  {
   return new Rectangle(point, size);
  }

  private Rectangle SizeToRectangle(PointF point, Size size)
  {
   return new Rectangle(new Point((int)point.X, (int)point.Y), size);
  }

  #region IDisposable Members

  public void Dispose()
  {
   if (_lockedBitmap != null)
   {
    _source.UnlockBits(_lockedBitmap);
    _lockedBitmap = null;
   }
  }

  #endregion
 }

How does this get you nearer to 3D vision? hint: red-blue differencing of the result...

Silverlight Eyes (Seyes) : eyes follow the mouse in Silverlight (aka xeyes)

Silverlight Eyes (Seyes) watches what you do and reports it to the Boss:



This code uses two main mechanism to achieve the effect

1. It uses Polar coordinates to calculate where the focal point is relative to the centre of each eyeball. (Often the problem people look for as finding the angle between two points). The code uses the angle to calculate what direction the pupil should be facing, and uses the radius (or distance) to proportionally move the pupil away from the centre.


2. It uses clipping to ensure that the pupil is always within the eye ball, which is accomplished by grouping the shapes into a canvas with a clipping region the same size of the eyeball.

The code also allows the user to visualize the calculations taking place by checking/unchecking the checkbox in the top left hand corner. Checking this shows the focal point, the lines of sight from each eye and the key values at the top (angle, distance, start point, end point)

There are a number of implementations of eye movement on the web, however a lot of them do not proportionally move the pupil away from the centre, this means the pupil in these animations are generally stuck at the edge of the eye. If you play with the application you will see that a proportional movement is more realistic - human brains are very good at calculating what other people are looking at based on the pupil positions.

Clipping can be seen in this picture:


The Xaml is as expected with the complication of grouping the parts of the eye together using a Canvas, with a Canvas.Clip to stop drawing outside of the eyeball:

<UserControl x:Class="Eyes.Page"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    >
    <Canvas x:Name="LayoutRoot" Background="White" MouseMove="LayoutRoot_MouseMove" Cursor="Arrow">
    <TextBlock Name="TextDebug" Canvas.Left="10" Canvas.Top="0" Width="1000" Height="100" />
    <CheckBox Name="ShowCalculation" Canvas.Left="0" Canvas.Top="0"
           Checked="CheckBox_Changed" Unchecked="CheckBox_Changed" Canvas.ZIndex="7"
           Cursor="Arrow"/>

    <Ellipse Canvas.Left="300" Canvas.Top="100" Width="230" Height="300" Stroke="Black" Name="Face" Fill="WhiteSmoke" Canvas.ZIndex="1"/>

    <Canvas Canvas.Left="340" Canvas.Top="200" Canvas.ZIndex="2" >
      <Ellipse Canvas.Left="0" Canvas.Top="0" Width="60" Height="40" Stroke="Black" Name="LeftEye" Fill="White" Canvas.ZIndex="2" />
      <Ellipse Canvas.Left="10" Canvas.Top="10" Width="20" Height="20" Fill="Green" Name="LeftPupil" Canvas.ZIndex="2" />
      <Canvas.Clip>
        <EllipseGeometry Center="30,20" RadiusX="30" RadiusY="20" />
      </Canvas.Clip>
    </Canvas>

    <Canvas Canvas.Left="420" Canvas.Top="200" Canvas.ZIndex="2" >
      <Ellipse Canvas.Left="0" Canvas.Top="0" Width="60" Height="40" Stroke="Black" Name="RightEye" Fill="White"  Canvas.ZIndex="2" />
      <Ellipse Canvas.Left="10" Canvas.Top="10" Width="20" Height="20" Fill="Green" Name="RightPupil" Canvas.ZIndex="2" />
      <Canvas.Clip>
        <EllipseGeometry Center="30,20" RadiusX="30" RadiusY="20" />
      </Canvas.Clip>
    </Canvas>

    <Ellipse Canvas.Left="395" Canvas.Top="270" Width="30" Height="40" Stroke="Black" Name="Nose" Fill="DarkGray" Canvas.ZIndex="1"/>

    <Ellipse Canvas.Left="380" Canvas.Top="320" Width="60" Height="30" Stroke="Black" Name="Mouth" Fill="DarkGray" Canvas.ZIndex="1"/>

    <Ellipse Canvas.Left="10" Canvas.Top="60" Width="10" Height="10" Fill="Red" Name="FocusPoint" Canvas.ZIndex="6" />
    <Line Stroke="Red" Name="LineSightLeft"  Canvas.ZIndex="5" StrokeDashArray="2 5" />
    <Line Stroke="Red" Name="LineSightRight"  Canvas.ZIndex="5" StrokeDashArray="2 5" />
  </Canvas>
</UserControl>

The code behind simply responds to the mouse moving, for each eye calculates the polar coordinates and then moves the eyeball proportionally towards the mouse:

public partial class Page : UserControl
{
 bool showCalcs;

 public Page()
 {
  InitializeComponent();

  // default to showing calculations
  ShowCalculation.IsChecked = this.showCalcs = true;
 }

 private void LayoutRoot_MouseMove(object sender, MouseEventArgs e)
 {
  // update UI based on cursor location
  Point cursorPoint = e.GetPosition(LayoutRoot);

  UpdateUI(cursorPoint);
 }

 private void UpdateUI(Point cursorPoint)
 {
  // calculate the location of the focus based using the center of the circle
  Point centerPoint = new Point(cursorPoint.X, cursorPoint.Y);

  if (this.showCalcs)
   FocusPoint.PointSet(new Point(cursorPoint.X - (FocusPoint.Width / 2), cursorPoint.Y - (FocusPoint.Height / 2)));

  // focus eyes on point
  FocusEye(centerPoint, LeftEye, LeftPupil, LineSightLeft);
  FocusEye(centerPoint, RightEye, RightPupil, LineSightRight);
 }

 private void FocusEye(Point pointFocus, Ellipse eye, Ellipse pupil, Line sight)
 {
  // the centre of the eye needs to be converted to a global coordinate
  // which it's parent has
  Point pointStart = new Point(eye.Parent.PointGet().X + (eye.Width / 2), eye.Parent.PointGet().Y + (eye.Height / 2));
  Point pointEnd = pointFocus;
  
  // cartesian to polar conversion
  double angle = Trigonometry.CalculateAngleInDegreesBetween(pointStart, pointEnd);
  double distance = Trigonometry.CalculateDistanceBetween(pointStart, pointEnd);

  if (this.showCalcs)
   TextDebug.Text = String.Format("  {2:N1}deg  {3:N1}    {0}    {1}", new object[] { pointStart, pointEnd, angle, distance });

  // maximum focal distance
  const double MAX_DISTANCE = 500d;

  // the eye will move away from the center
  // of the eye ball proportionally to its maxim view
  // reduced by π to keep some of it visible
  double pupilDistance = (eye.Width / Math.PI) * (distance / MAX_DISTANCE);

  // polar to cartesian conversion
  Point pointPupilOffset = Trigonometry.ConvertPolarToPoint(pupilDistance, angle);

  // adjust for centre of eyeball and pupil
  Point pupilLocation = new Point(
    (eye.Width / 2) + pointPupilOffset.X - (pupil.Width / 2),
    (eye.Height / 2) - pointPupilOffset.Y - (pupil.Height / 2));

  pupil.PointSet(pupilLocation);

  if (this.showCalcs)
  {
   sight.X1 = pointStart.X;
   sight.Y1 = pointStart.Y;
   sight.X2 = pointEnd.X;
   sight.Y2 = pointEnd.Y;
  }
 }

 private void CheckBox_Changed(object sender, RoutedEventArgs e)
 {
  // show calculations visually
  this.showCalcs = ShowCalculation.IsChecked ?? false;

  Visibility visibility = (this.showCalcs) ? Visibility.Visible : Visibility.Collapsed;

  FocusPoint.Visibility = visibility;
  LineSightLeft.Visibility = visibility;
  LineSightRight.Visibility = visibility;
  TextDebug.Visibility = visibility;

  if (this.showCalcs)
   LayoutRoot.Cursor = Cursors.None;
  else
   LayoutRoot.Cursor = Cursors.Arrow;
 }
}

The code uses some extension methods, as I started doing in my previous blog entry:

public static class XamlExtensions
{
 public static Point PointGet(this DependencyObject sobj)
 {
  Point point = new Point((double)sobj.GetValue(Canvas.LeftProperty), (double)sobj.GetValue(Canvas.TopProperty));

  return point;
 }

 public static void PointSet(this DependencyObject shape, Point newLocation)
 {
  shape.SetValue(Canvas.LeftProperty, newLocation.X);
  shape.SetValue(Canvas.TopProperty, newLocation.Y);
 }

 public static Point GetCenterPoint(this Shape shape)
 {
  Point location = shape.PointGet();

  Point centerPoint = new Point(location.X + (shape.Width / 2), location.Y + (shape.Height / 2));

  return centerPoint;
 }
}

and a helper class for doing the trig:

public class Trigonometry
{
 public static double DegreesToRadians(double angle)
 {
  return ((angle * Math.PI) / 180f);
 }

 public static double RadiansToDegrees(double angle)
 {
  return ((angle * 180) / Math.PI);
 }

 public static double CalculateAngleInDegreesBetween(Point pointStart, Point pointEnd)
 {
  // difference between points on Y-Axis
  // Silverlight Y axis needs to be inverted of cartesian coordinates
  double dy = -(pointEnd.Y - pointStart.Y);
  // difference between points on X-Axis
  double dx = (pointEnd.X - pointStart.X);

  // angle of the vector defined by dy, dx
  double angleRadians = Math.Atan2(dy, dx);

  // results in (-π,π], which needs mapping to [0,2π) by adding 2π to negative 
  if (angleRadians < 0)
   angleRadians += 2 * Math.PI;

  double angleDegrees = RadiansToDegrees(angleRadians);

  return angleDegrees;
 }

 public static double CalculateDistanceBetween(Point pointStart, Point pointEnd)
 {
  double dySquared = Math.Pow(pointEnd.Y - pointStart.Y, 2);
  double dxSquared = Math.Pow(pointEnd.X - pointStart.X, 2);

  double distance = Math.Sqrt(dySquared + dxSquared);

  return distance;
 }


 internal static Point ConvertPolarToPoint(double radius, double angleDegrees)
 {
  double angleRadians = DegreesToRadians(angleDegrees);
  
  double x = radius * Math.Cos(angleRadians);
  double y = radius * Math.Sin(angleRadians);

  return new Point(x, y);
 }
}