Exploring Firebase MLKit on Android: Barcode Scanning (Part Three)

Facebook Twitter Reddit

At Google I/O this year we saw the introduction of Firebase MLKit, a part of the Firebase suite that intends to give our apps the ability to support intelligent features with more ease. With this comes the barcode scanning feature, giving us the ability to scan barcodes and QR codes to retrieve data from the real world and manipulate it inside of our applications. In this post I want to dive into how we can implement this feature into our applications.

When I first started looking into Ml Kit I’ll admit that the bar-code scanning feature was the one I was least excited about — but after having a chance to read up on it and play with it I began to discover that it’s a really well thought out part of the Ml Kit suite. I actually learnt quite a bit about bar-codes and to begin with, the bar-code scanning part of ML Kit supports the following formats of bar-code formats:

I’m going to be honest here and admit that I originally had no idea there were so many different types of barcodes (I’m certain this isn’t them all either). If you need to support some (or all) of these, then the Bar-code scanning part of ML Kit will handle all of this for you.

Now, it is probably time to answer that one question that might be on peoples minds, but why? As primitive and simple that barcodes seem, that is the point! Barcodes are accessible but many — I myself in the past have joked about “Oh not another QR code!”, but the great thing about these is that they don’t require any special or powerful hardware to read, are platform independent and can be read by a number of different means (such as camera, barcode scanner etc). This makes barcodes accessible to many different users and devices around the world, regardless of how low-end the device is as all we require is a camera or some means to read the barcode data.

This is great news for your app as it Firebase Ml Kit now makes this functionality even easier. If you’re already using a third-party SDK or requiring your users to use another app for barcode scanning functionality, then you can now make use of Firebase to do this for you.

Before we can start using the barcode feature of MLKit, we need to begin by adding the dependency to our project level build.gradle file:

implementation 'com.google.firebase:firebase-ml-vision:16.0.0'

Now, if you want the barcode scanning part of MLKit to be downloaded at the point of application install, then you can add the following snippet within the application tags of your manifest file. Otherwise, the barcode scanning part of the MLKit library will be downloaded at the point where it is required within your application.

<meta-data
      android:name="com.google.firebase.ml.vision.DEPENDENCIES"
      android:value="barcode" />

Now we‘ve done the above, we’re ready to add bar-code scanning to our application. We can retrieve an instance of the VisionBarcodeDetector class which handles the recognition process for us:

val detector = FirebaseVision.getInstance().visionBarcodeDetector

When it comes to this recognition, we can optionally configure the barcode formats that we wish to be supported in the recognition process. This is done via the means of a FirebaseVisionBarcodeDetectorOptions instance. We can create a new instance of this using the class builder:

val options = FirebaseVisionBarcodeDetectorOptions.Builder()
        .setBarcodeFormats(
                FirebaseVisionBarcode.FORMAT_QR_CODE,
                FirebaseVisionBarcode.FORMAT_AZTEC)
        .build()

Now that we have this options defined, we can the use the get function of our FirebaseVision instance passing in our options instance:

val detector = FirebaseVision.getInstance().getVisionBarcodeDetector(options)

Now that we have our options built, we can go ahead and make use of them in our recognition flow. We want to use these options to create an instance of a FirebaseVisionImage — this is a class which holds our image data ready for the recognition process. Now, we need this instance before we can perform any form of recognition and in order to create an instance of this we need to use our image data — this can be done in one of five ways:

Bitmap

To begin with, we can create this instance of a FirebaseVisionImage using an instance of a Bitmap. We can do so by passing an upright bitmap into the fromBitmap() function — this will give us back a FirebaseVisionImage

val image = FirebaseVisionImage.fromBitmap(bitmap);

media.Image

We can also do so using a media.Image instance — this may be when capturing an image from the devices camera. When doing so we must pass the instance of this image as well as the rotation of it, so this must be calculated prior to calling the fromMediaImage() function.

val image = FirebaseVisionImage.fromMediaImage(mediaImage,    
                rotation);

ByteBuffer

An instance can also be created using a ByteBuffer. To do so though we must first create an instance of a FirebaseVisionImageMetadata. This contains the data required to construct the vision image, such as the rotation and measurements.

FirebaseVisionImageMetadata metadata = new 
    FirebaseVisionImageMetadata.Builder()
        .setWidth(1280)
        .setHeight(720)
        .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21)
        .setRotation(rotation)
        .build();

We can then pass this along with our ByteBuffer to create the instance:

val image = FirebaseVisionImage.fromByteBuffer(buffer, metadata);

ByteArray

Creating an image from a ByteArray behaves in the same way as a ByteBuffer except we must using the fromByteArray() function instead:

val image = FirebaseVisionImage.fromByteArray(byteArray, metadata);

File

A vision image instance can be created from a file by calling the fromFilePath() function with a context and desired URI.

val image: FirebaseVisionImage?
try {
    image = FirebaseVisionImage.fromFilePath(context, uri);
} catch (IOException e) {
    e.printStackTrace();
}

The approach which you use to retrieve the FirebaseVisionImage instance will depend on your application and how you are working with images. However you do so, at this point you should have access to an instance of the FirebaseVisionImage class. At this point our VisionBarcodeDetector is also configured and ready to go, so now we can go ahead and use this FirebaseBarcodeDetector instance to detect the barcodes in our image. This can be done by calling the detectInImage() function, passing in our FirebaseVisionImage instance:

detector.detectInImage(image)
    .addOnSuccessListener {
// Task succeeded!
        for (barcode in it) {
            // Do something with barcode
        }
}
.addOnFailureListener {
// Task failed with an exception
}

Now, if this call succeeds then we will be given a list of FirebaseVisionBarcode instances. If no barcodes have been detected then this will be empty, so you need to handle this if this situation occurs. Otherwise, we have access to a collection of barcodes that we now need to do something with. For each FirebaseVisionBarcode instance, we have access to a collection of properties that we can use here:

getBoundingBox() — Returns a Rect instance that contains the bounding box for the recognised barcode.
getCornerPoints() — Returns the coordinates for each corner of the barcode.
getRawValue() — Returns the barcode value in its raw format
getDisplayValue() — Returns the barcode value in a readable format
getValueType() — Returns the format type of the barcode

The most important part of the properties above which we get back is the value type of the barcode. This gives us the format of the barcode and then depicts the type of data that we can get back from the FirebaseVisionBarcode instance. There are 9 different data types (TYPE_)which barcodes can be represented as, and each of these will have a data object attached to it containing information about the data from the barcode:

Let’s have a look at what each of these will have available when read from a barcode and how we can access them from our barcode instance:

FirebaseVisionBarcode.CalendarEvent (TYPE_CALENDAR_EVENT)

getDescription() — Returns a description of the event
getStart()- Returns an instance of FirebaseVisionBarcode.CalendarDateTime for the start of the event
getEnd() — Returns an instance of FirebaseVisionBarcode.CalendarDateTime for the end of the event
getLocation() — Returns the location of the event
getOrganizer() — Returns the organiser of the event
getStatus() — Returns the status of the event
getSummary() — Returns a summary for the ven

when (valueType) {
    FirebaseVisionBarcode.TYPE_CALENDAR_EVENT -> {
        val description = barcode.calendarEvent?.description
val start = barcode.calendarEvent?.start
val end = barcode.calendarEvent?.end
val organizer = barcode.calendarEvent?.organizer
val summary = barcode.calendarEvent?.summary
val status = barcode.calendarEvent?.status
val location = barcode.calendarEvent?.location
}
}

FirebaseVisionBarcode.ContactInfo (TYPE_CONTACT_INFO)

getAddresses() — Returns a list of FirebaseVisionBarcode.Address instances for the contact
getEmails() — Returns a list of FirebaseVisionBarcode.Email for the contact
getName() — Returns an instance of FirebaseVisionBarcode.PersonName for the contact
getOrganization() — Returns the contacts organization
getPhones() — Returns a list of FirebaseVisionBarcode.Phone for the contact
getTitle() — Returns the contacts title
getUrls() — Returns an array of Urls for the contact

when (valueType) {
    FirebaseVisionBarcode.TYPE_CONTACT_INFO -> {
        val addresses = barcode.contactInfo?.addresses
val emails = barcode.contactInfo?.emails
val phones = barcode.contactInfo?.phones
val names = barcode.contactInfo?.name
val organization = barcode.contactInfo?.organization
val title = barcode.contactInfo?.title
val urls = barcode.contactInfo?.urls
}
}

FirebaseVisionBarcode.DriverLicense (TYPE_DRIVER_LICENSE)

getAddressCity() — Returns the city for the license address
getAddressState() — Returns the state for the license address
getAddressStreet() — Returns the street for the license address
getAddressZip() — Returns the zip for the license address
getBirthDate() — Returns the date of birth for the license holder
getDocumentType()- Returns the document type (Driver License or ID card)
getExpiryDate() — Returns the expiry date for the licence
getFirstName() — Returns the first name of the license holder
getMiddleName() — Returns the middle name of the license holder
getLastName() — Returns the last name of the license holder
getGender() — Returns the gender of the license holder
getIssueDate() — Returns the issue date of the license
getIssuingCountry() — Returns the issuing country of the license
getLicenseNumber() — Returns the license number

when (valueType) {
    FirebaseVisionBarcode.TYPE_DRIVER_LICENSE -> {
        val city = barcode.driverLicense?.addressCity
val state = barcode.driverLicense?.addressState
val street = barcode.driverLicense?.addressStreet
val zip = barcode.driverLicense?.addressZip
val birthDate = barcode.driverLicense?.birthDate
val document = barcode.driverLicense?.documentType
val expiry = barcode.driverLicense?.expiryDate
val firstName = barcode.driverLicense?.firstName
val middleName = barcode.driverLicense?.middleName
val lastName = barcode.driverLicense?.lastName
val gender = barcode.driverLicense?.gender
val issueDate = barcode.driverLicense?.issueDate
val issueCountry = barcode.driverLicense?.issuingCountry
val licenseNumber = barcode.driverLicense?.licenseNumber
}
}

FirebaseVisionBarcode.Email (TYPE_EMAIL)

getAddress() — Returns the email address used for the email
getBody() — Returns the body of the email
getSubject() — Returns the subject of the email
getType() — Returns the type of email (HOME, WORK, UNKNOWN)

when (valueType) {
    FirebaseVisionBarcode.TYPE_EMAIL -> {
        val type = barcode.email?.type
val address = barcode.email?.address
val body = barcode.email?.body
val subject = barcode.email?.subject
}
}

FirebaseVisionBarcode.GeoPoint (TYPE_GEO)

getLat() — Returns the latitude for the corresponding geo point
getLng() — Returns the longitude for the corresponding geo point

when (valueType) {
    FirebaseVisionBarcode.TYPE_GEO -> {
        val lat = barcode.geoPoint?.lat
val lng = barcode.geoPoint?.lng
}
}

FirebaseVisionBarcode.Phone (TYPE_PHONE)

getNumber() — Returns the number for the corresponding phone instance
getType() — Returns the type of phone (FAX, HOME, MOBILE, WORK, UNKNOWN)

when (valueType) {
    FirebaseVisionBarcode.TYPE_PHONE -> {
        val number = barcode.phone?.number
val type = barcode.phone?.type
}
}

FirebaseVisionBarcode.Sms (TYPE_SMS)

getMessage() — Returns the message for the given SMS
getPhoneNumber() — Returns the phone number associated with the given SMS

when (valueType) {
    FirebaseVisionBarcode.TYPE_SMS -> {
        val message = barcode.sms?.message
val number = barcode.sms?.phoneNumber
}
}

FirebaseVisionBarcode.UrlBookmark (TYPE_URL)

getTitle() — Returns the title for the given bookmark
getUrl() — Returns the URL for the given bookmark

when (valueType) {
    FirebaseVisionBarcode.TYPE_URL -> {
        val title = barcode.url?.title
val url = barcode.url?.url
}
}

FirebaseVisionBarcode.Wifi (TYPE_WIFI)

getEncryptionType() — Returns the encryption type (OPEN, WEP or WPA) for the given network
getPassword() — Returns the password for the given WIFI network
getSsid() — Returns the SSID for the given WIFI network

when (valueType) {
    FirebaseVisionBarcode.TYPE_WIFI -> {
        val ssid = barcode.wifi?.ssid
val password = barcode.wifi?.password
val type = barcode.wifi?.encryptionType
}
}

Other than the data retrieved from the barcode in relation to the barcode format, as previously mentioned we also have access things such as the barcode bounds and corners. In the image below here, I’ve taken the boundingBox property of the FirebaseVisionBarcode instance that was returned by the scanning request. Using the properties of the bounding box I’ve then been able to draw these bounds onto the image of the barcode which I originally passed into the scanner. You could use these bounds to do a similar thing in your applications — doing so allows your user to know that the barcode that they are trying to scan has been recognised successfully by the scanning operation that is currently taking place in your app.

Sometimes though, you may not want to show an entire bounding box around the whole of the barcode — in situations when this is the case the FirebaseVisionBarcode also returns us a cornerPoints property. In this case you can simply use these corner points to paint the points onto the canvas. You could also use this to draw some form of custom outline on the barcode that maybe only extends to a certain point along the barcode.

Wow, Barcode Scanning with MLKit is pretty neat isn’t it! From this article I hope you have been able to see just how much simpler Firebase have made this scanning process for our applications. What are your thoughts? If you have any comments or questions, please do reach out 🙂

Tweets by hitherejoe

Joe Birch

Speaker, Educator and GDE for Android