In-Depth
Plug in to Mono for Android for Lights, Camera and Video Apps
Multimedia is expected in today's mobile applications, and the fun isn't only for smartphone and tablet users. Mono for Android, based on the Mono project's open source implementation of the Microsoft .NET Framework, can help you take advantage of Android Camera APIs in Visual Studio.
I don't know about you, but I'm amazed at what you can do with the multimedia in today's mobile devices. I grew up playing "Pong" and remember when games had to fit into big boxes with green screens and ASCII art. About 10 years ago, I had to create an interpreter for the graphics processor in the IBM AS/400 systems (iSeries now) to generate graphics in a screen-scraping application. How did we survive those days?
Now, mobile devices allow you to take pictures, record audio and video, and send it all to friends and relatives in minutes. You're seeing this happen all over the world, even in uprisings, as people compete to get their stories out.
Many people are confused by what multimedia actually is (including me), but for our purposes, I'll look at how .NET developers can use the Xamarin Mono for Android plug-in with Visual Studio 2010 to create Android applications that use pictures and video. If you're new to Mono for Android, check out my tutorial, "Introduction to MonoDroid".
While Mono for Android has definitely grown and matured since then, many of the basics are still applicable. However, the Mono garbage collector (GC) is not quite as advanced as the GC in the Microsoft .NET Framework. Sometimes, you'll need to help the GC along by calling GC.Collect at an appropriate part of your application.
There are a large number of Android devices in the marketplace. What works on one may not work properly on another. The code in this article was tested and validated against Mono for Android 4.0.x. Testing was done on two devices: an HTC EVO 4G running Android 2.3 (code-named "Gingerbread"), and a Motorola Xoom tablet running Android 4.0.3 (code-named "Ice Cream Sandwich").
I've found that if an app will work on my HTC EVO 4G and my Motorola Xoom, it typically works across many devices.
Asking for Permissions
Mobile applications shouldn't just allow complete access to the hardware and other software in a device. With Android, developers must request access to various features on the device, such as the camera. To ask for access, you must include an AndroidManifest.xml file in your application. Mono for Android includes a default AndroidManifest.xml file based on the various attributes of the activities. It's not something you see in the file system, but if you unzip the output .apk file, you'll find an AndroidManifest.xml file. Unfortunately, the resulting file is in binary, so it's rather hard to read, but it's there.
To set permissions and use other features of Android (basically, all of the cool stuff), you must create an explicit AndroidManifest.xml file for your Mono for Android project. This is done by going to the properties of the project, selecting the Android Manifest tab and then creating the AndroidManfest.xml file, as shown in Figure 1.
[Click on image for larger view.] |
| Figure 1. To set permissions and use other features of the Android OS, you must create an explicit AndroidManifest.xml file for your Mono for Android project. |
Once this is done, an AndroidManifest.xml file is created in the Properties folder of the project. This is shown in Figure 2.
[Click on image for larger view.] |
| Figure 2. An AndroidManifest.xml file is created in the Properties folder of the project. |
Now that you have an AndroidManifest.xml file, you can edit it in one of two ways. Mono for Android has an editor for setting the permissions. You can add various permissions to use features in the phone, specify the minimum supported version of Android, identify what features are required in the device for the application and stipulate other items. For example, an application that's writing images to the device's file system will probably need android.permission.WRITE_EXTERNAL_STORAGE. An application that records audio will need android.permission.RECORD_AUDIO. If the application needs to embed GPS data into images, then it will require either adroid.permission.ACCESS_FINE_LOCATION or adroid.permission.ACCESS_COARSE_LOCATION.
Listing 1 shows the XML code for the AndroidManifest file. The application requests the Camera, Internet and Record_Audio permissions so it can use these features in the device.
Once you have permissions, you need to document the features you'll need to have on the device. In Figure 3, the file documents that the application will use the camera.
[Click on image for larger view.] |
| Figure 3. The Camera permission is requested by the application. |
Displaying Images
Pictures allow people to process a story much better than just text alone. Android has the ability to display pictures as well as create them with a camera. Displaying images is fairly simple in Android, but there's one situation in which you need to be careful. You can get pictures in Android in three ways. First, images can be built into an application. The advantage of this method is that the image is basically guaranteed to be available. It's fairly simple programmatically because you can reference the image with the code Resource.Drawable.PictureName, so that you can access it via IntelliSense. An example call is:
iv.SetImageResource(Resource.Drawable.AndroidLocalImage);
Note that the resource is created for you in the Android.Designer.cs file. There's also a limit on local images stored as resources. This scenario is good for standard images that are necessary in your application. There's no need to go outside the device, so you won't be required to use a network, which is often spotty at best.
Second, images can be on the device, but not built into the application. In this scenario, you don't have IntelliSense access to the images. To load an image off the device, you call an ImageView SetImageURI method and pass in a local Android.Net.Uri path to the file.
Third, images can be pulled from a location outside the device, typically over the Internet.
Mobile networks are not as reliable as wired networks. Add in that Android will close applications that have locked the UI thread for too long, and you can have problems if you attempt to access this on the UI thread. Think about displaying a Twitter search in a ListView. Instead of displaying one picture from outside the device, now you have to display up to 20 images. A better strategy is to load an image asynchronously on a background thread. An example call to start loading the image off of a network resource is shown in Listing 2.
In Listing 2, the button's Click event starts a ThreadPool that will load images from a remote resource. Once the thread is started, the image is downloaded into a bitmap object in the local device's memory, and then loaded in the application. Note that you have to use RunOnUIThread to perform the loading on the UI control because it's running on a background thread.
Using Native Camera Functionality
The APIs in Android allow developers to integrate with cameras in different ways. The question is whether you want to use the existing functionality in your Android phone or build custom functionality.
Most applications can use the existing camera application. Creating an Intent is an easy and quick way to enable taking pictures from within your app. Once the Camera Intent is fired, the native camera application loads and a user can "point and shoot"; then program control is returned to your application. Here's how to use a Camera Intent to take pictures in your application:
- Create a Camera Intent. This Camera Intent will have the action Adroid. Prvider.MediaStore.ActionImageCapture. Inside this Intent, pass in the file where you want to save the image. Some Android devices require the image save file; some do not. I've found that passingthis value in results in more devices working than if I didn't take this step.
- To start the Camera Intent, call the method StartActivityForResult.
This will start the camera application and allow the user to take a picture.
- Once a user is done taking a picture and execution passes back to your program, you'll override the OnActivityResult method to receive the callback, get the picture and then finish any processing there.
The code to handle this processing is shown in Listing 3.
Out of this Listing 3 code, you should see the sequence of pictures shown in Figure 4.
Figure 4 (A) is the image shown in the Camera Activity before the picture is actually taken. Figure 4 (B) is the same image after it has been selected in a Camera Intent and is waiting for the user to select. The Camera Activity is asking if the user is "Done," which sends execution back to your application. There's also an option to return to the Camera Activity and take a different picture. Figure 4 (C) shows the original application displaying the returned image.
[Click on image for larger view.] |
| Figure 4. (A) The image is shown in the Camera Activity before the picture is actually taken. (B) The same image after it has been selected in a Camera Intent and is waiting for the user to select.
(C) The image is returned and displayed back in the original application. |
Note that when a program interfaces with the built-in Activities, they have the built-in security permissions.
Taking Your Own Pictures
Depending on the built-in Activities, and communicating with Intents, is the way to go for more than 90 percent of applications. Other apps can implement their own functionality to integrate with the camera.
The Mono for Android framework provides access to the Android Camera APIs via the Android.Hardware.Camera class. The Camera class is used to set image capture values, perform preview operations, take pictures and retrieve frames for encoding video. The Camera class will interface with the Android Camera service, which manages communications with the camera hardware.
Follow this general sequence to take pictures in your app using a custom Camera:
- Create an instance of the Camera by calling Open.
- Get a set of the default settings by calling the Camera's GetParameters method.
- Modify any parameters as necessary.
- Save the new parameters back to the Camera by calling SetParameters.
- If necessary, call SetDisplayOrientation. This will set the clockwise rotation of the preview display.
- To provide a preview surface, pass a fully initialized SurfaceHolder into the method SetPreviewDisplay.
- To start previewing, call StartPreview. Previewing must be started before a picture can be taken.
- A picture can be taken by calling TakePicture. The image data will be provided in the necessary callbacks.
- After taking a picture, previewing will stop. To start previewing again, call StartPreview.
- To stop updating the preview surface, call StopPreview.
- To allow use of a camera in other applications, make sure that Release is called as appropriate. For example, Release is invoked in OnPause, and open is called in OnResume.
The following interfaces can be implemented from the Camera class:
- Android.Hardware.Camera.IAutoFocusCallback This interface includes the OnAutoFocus method, which can be implemented to notify the application that the camera has completed auto-focus. According to the Android document, devices that don't support auto-focus will receive a "face" callback to this interface. If an application requires auto-focus support in hardware, it should declare this support in the AndroidManifest.xml file. This support is declared by placing the android.hardware.camera.autofocus feature in the <uses-feature> tag.
- Android.Hardware.Camera.IErrorCallback This interface is error notification from the camera. The OnError method is called if there's an error.
- Android.Hardware.Camera.IFaceDetectionListener This interface is used for face detection in the preview frame. The OnFaceDetection method is called when a face is detected.
- Android.Hardware.Camera.IOnZoomChangeListener This interface is used to handle zoom changes during a smooth zoom operation. The method associated with this interface is OnZoomChangeListener.
- Android.Hardware.Camera.IPictureCallback This interface is used to get the image data from a photo capture. The method associated with this interface is OnPictureTaken.
- Android.Hardware.Camera.IPreviewCallback This interface is used to deliver preview frames. The method associated with the interface is OnPreviewFrame.
- Android.Hardware.Camera.ShutterCallback This interface is used to notify the application that an image capture occurred at that particular moment.