Month: May 2017

Sample: Custom Vision in Xamarin.Forms with Azure customvision.ai

When I saw Azure Cognitive Services’ customvision.ai session at Build, I needed to play with it as soon as possible. I had dreamed of the day when this level of abstraction made custom vision readily available without needing to rebuild and tune vast computer vision systems full of magic constants and poorly performing algorithms. With CustomVision.ai, it’s simple: you upload several photos of something and tag them, enabling the prediction service to identify with a great level of accuracy whether or not any of your tags appear in a sample photo. You can also train the system in realtime with new data any time you want.

customvision

I build an app called PetID which allows me to snap a photo (camera view reused from the Moments sample – cheers Pierce!). The photo is then sent as a stream to the Custom Vision Prediction endpoint. Then, magic in the cloud happens, and out pops a list of ImageTagPredictions. This list pairs up a tag and the probability that the supplied image contains the tag based on the training data you provided. For example, I uploaded about 20 photos each of my 3 pets, tagged them, and trained the system. I did all this by clicking around the UI. That’s it!

 

 

 

 

There are two NuGet packages for customvision.ai but they are not available as PCLs; they are autogenerated wrappers around the APIs which also drive the website. This means I had to do a little bit of extra work to get it to work with Forms, namely, a custom renderer for the iOS camera view and a DependencyService to abstract the prediction endpoints since I can’t call them from a Forms PCL.

Using this sample

Download the sample here: https://github.com/danwaters/PetID

You’ll need to set up a Training Key and Prediction Key at http//customvision.ai and train a few images to make this work. Then, open APIKeys.cs and add your own keys.

The code is generic enough that it should work with any prediction endpoint, not just my pets.

Conclusion

Extremely powerful vision recognition and classification is now at your fingertips in Azure. There are dozens of applications for this technology – what can you build with it?