When I first started looking into Ml Kit I’ll admit that the bar-code scanning feature was the one I was least excited about — but after having a chance to read up on it and play with it I began to discover that it’s a really well thought out part of the Ml Kit suite. I actually learnt quite a bit about bar-codes and to begin with, the bar-code scanning part of ML Kit supports the following formats of bar-code formats:
I’m going to be honest here and admit that I originally had no idea there were so many different types of barcodes (I’m certain this isn’t them all either). If you need to support some (or all) of these, then the Bar-code scanning part of ML Kit will handle all of this for you.
Now, it is probably time to answer that one question that might be on peoples minds, but why? As primitive and simple that barcodes seem, that is the point! Barcodes are accessible but many — I myself in the past have joked about “Oh not another QR code!”, but the great thing about these is that they don’t require any special or powerful hardware to read, are platform independent and can be read by a number of different means (such as camera, barcode scanner etc). This makes barcodes accessible to many different users and devices around the world, regardless of how low-end the device is as all we require is a camera or some means to read the barcode data.
This is great news for your app as it Firebase Ml Kit now makes this functionality even easier. If you’re already using a third-party SDK or requiring your users to use another app for barcode scanning functionality, then you can now make use of Firebase to do this for you.
Before we can start using the barcode feature of MLKit, we need to begin by adding the dependency to our project level build.gradle file:
implementation 'com.google.firebase:firebase-ml-vision:16.0.0'
Now, if you want the barcode scanning part of MLKit to be downloaded at the point of application install, then you can add the following snippet within the application tags of your manifest file. Otherwise, the barcode scanning part of the MLKit library will be downloaded at the point where it is required within your application.
<meta-data android:name="com.google.firebase.ml.vision.DEPENDENCIES" android:value="barcode" />
Now we‘ve done the above, we’re ready to add bar-code scanning to our application. We can retrieve an instance of the VisionBarcodeDetector class which handles the recognition process for us:
val detector = FirebaseVision.getInstance().visionBarcodeDetector
When it comes to this recognition, we can optionally configure the barcode formats that we wish to be supported in the recognition process. This is done via the means of a FirebaseVisionBarcodeDetectorOptions instance. We can create a new instance of this using the class builder:
val options = FirebaseVisionBarcodeDetectorOptions.Builder() .setBarcodeFormats( FirebaseVisionBarcode.FORMAT_QR_CODE, FirebaseVisionBarcode.FORMAT_AZTEC) .build()
Now that we have this options defined, we can the use the get function of our FirebaseVision instance passing in our options instance:
val detector = FirebaseVision.getInstance().getVisionBarcodeDetector(options)
Now that we have our options built, we can go ahead and make use of them in our recognition flow. We want to use these options to create an instance of a FirebaseVisionImage — this is a class which holds our image data ready for the recognition process. Now, we need this instance before we can perform any form of recognition and in order to create an instance of this we need to use our image data — this can be done in one of five ways:
Bitmap
To begin with, we can create this instance of a FirebaseVisionImage using an instance of a Bitmap. We can do so by passing an upright bitmap into the fromBitmap() function — this will give us back a FirebaseVisionImage
val image = FirebaseVisionImage.fromBitmap(bitmap);
media.Image
We can also do so using a media.Image instance — this may be when capturing an image from the devices camera. When doing so we must pass the instance of this image as well as the rotation of it, so this must be calculated prior to calling the fromMediaImage() function.
val image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation);
ByteBuffer
An instance can also be created using a ByteBuffer. To do so though we must first create an instance of a FirebaseVisionImageMetadata. This contains the data required to construct the vision image, such as the rotation and measurements.
FirebaseVisionImageMetadata metadata = new FirebaseVisionImageMetadata.Builder() .setWidth(1280) .setHeight(720) .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21) .setRotation(rotation) .build();
We can then pass this along with our ByteBuffer to create the instance:
val image = FirebaseVisionImage.fromByteBuffer(buffer, metadata);
ByteArray
Creating an image from a ByteArray behaves in the same way as a ByteBuffer except we must using the fromByteArray() function instead:
val image = FirebaseVisionImage.fromByteArray(byteArray, metadata);
File
A vision image instance can be created from a file by calling the fromFilePath() function with a context and desired URI.
val image: FirebaseVisionImage? try { image = FirebaseVisionImage.fromFilePath(context, uri); } catch (IOException e) { e.printStackTrace(); }
The approach which you use to retrieve the FirebaseVisionImage instance will depend on your application and how you are working with images. However you do so, at this point you should have access to an instance of the FirebaseVisionImage class. At this point our VisionBarcodeDetector is also configured and ready to go, so now we can go ahead and use this FirebaseBarcodeDetector instance to detect the barcodes in our image. This can be done by calling the detectInImage() function, passing in our FirebaseVisionImage instance:
detector.detectInImage(image) .addOnSuccessListener { // Task succeeded! for (barcode in it) { // Do something with barcode } } .addOnFailureListener { // Task failed with an exception }
Now, if this call succeeds then we will be given a list of FirebaseVisionBarcode instances. If no barcodes have been detected then this will be empty, so you need to handle this if this situation occurs. Otherwise, we have access to a collection of barcodes that we now need to do something with. For each FirebaseVisionBarcode instance, we have access to a collection of properties that we can use here:
- getBoundingBox() — Returns a Rect instance that contains the bounding box for the recognised barcode.
- getCornerPoints() — Returns the coordinates for each corner of the barcode.
- getRawValue() — Returns the barcode value in its raw format
- getDisplayValue() — Returns the barcode value in a readable format
- getValueType() — Returns the format type of the barcode
The most important part of the properties above which we get back is the value type of the barcode. This gives us the format of the barcode and then depicts the type of data that we can get back from the FirebaseVisionBarcode instance. There are 9 different data types (TYPE_)which barcodes can be represented as, and each of these will have a data object attached to it containing information about the data from the barcode:
Let’s have a look at what each of these will have available when read from a barcode and how we can access them from our barcode instance:
FirebaseVisionBarcode.CalendarEvent (TYPE_CALENDAR_EVENT)
- getDescription() — Returns a description of the event
- getStart()- Returns an instance of FirebaseVisionBarcode.CalendarDateTime for the start of the event
- getEnd() — Returns an instance of FirebaseVisionBarcode.CalendarDateTime for the end of the event
- getLocation() — Returns the location of the event
- getOrganizer() — Returns the organiser of the event
- getStatus() — Returns the status of the event
- getSummary() — Returns a summary for the ven
when (valueType) { FirebaseVisionBarcode.TYPE_CALENDAR_EVENT -> { val description = barcode.calendarEvent?.description val start = barcode.calendarEvent?.start val end = barcode.calendarEvent?.end val organizer = barcode.calendarEvent?.organizer val summary = barcode.calendarEvent?.summary val status = barcode.calendarEvent?.status val location = barcode.calendarEvent?.location } }
FirebaseVisionBarcode.ContactInfo (TYPE_CONTACT_INFO)
- getAddresses() — Returns a list of FirebaseVisionBarcode.Address instances for the contact
- getEmails() — Returns a list of FirebaseVisionBarcode.Email for the contact
- getName() — Returns an instance of FirebaseVisionBarcode.PersonName for the contact
- getOrganization() — Returns the contacts organization
- getPhones() — Returns a list of FirebaseVisionBarcode.Phone for the contact
- getTitle() — Returns the contacts title
- getUrls() — Returns an array of Urls for the contact
when (valueType) { FirebaseVisionBarcode.TYPE_CONTACT_INFO -> { val addresses = barcode.contactInfo?.addresses val emails = barcode.contactInfo?.emails val phones = barcode.contactInfo?.phones val names = barcode.contactInfo?.name val organization = barcode.contactInfo?.organization val title = barcode.contactInfo?.title val urls = barcode.contactInfo?.urls } }
FirebaseVisionBarcode.DriverLicense (TYPE_DRIVER_LICENSE)
- getAddressCity() — Returns the city for the license address
- getAddressState() — Returns the state for the license address
- getAddressStreet() — Returns the street for the license address
- getAddressZip() — Returns the zip for the license address
- getBirthDate() — Returns the date of birth for the license holder
- getDocumentType()- Returns the document type (Driver License or ID card)
- getExpiryDate() — Returns the expiry date for the licence
- getFirstName() — Returns the first name of the license holder
- getMiddleName() — Returns the middle name of the license holder
- getLastName() — Returns the last name of the license holder
- getGender() — Returns the gender of the license holder
- getIssueDate() — Returns the issue date of the license
- getIssuingCountry() — Returns the issuing country of the license
- getLicenseNumber() — Returns the license number
when (valueType) { FirebaseVisionBarcode.TYPE_DRIVER_LICENSE -> { val city = barcode.driverLicense?.addressCity val state = barcode.driverLicense?.addressState val street = barcode.driverLicense?.addressStreet val zip = barcode.driverLicense?.addressZip val birthDate = barcode.driverLicense?.birthDate val document = barcode.driverLicense?.documentType val expiry = barcode.driverLicense?.expiryDate val firstName = barcode.driverLicense?.firstName val middleName = barcode.driverLicense?.middleName val lastName = barcode.driverLicense?.lastName val gender = barcode.driverLicense?.gender val issueDate = barcode.driverLicense?.issueDate val issueCountry = barcode.driverLicense?.issuingCountry val licenseNumber = barcode.driverLicense?.licenseNumber } }
FirebaseVisionBarcode.Email (TYPE_EMAIL)
- getAddress() — Returns the email address used for the email
- getBody() — Returns the body of the email
- getSubject() — Returns the subject of the email
- getType() — Returns the type of email (HOME, WORK, UNKNOWN)
when (valueType) { FirebaseVisionBarcode.TYPE_EMAIL -> { val type = barcode.email?.type val address = barcode.email?.address val body = barcode.email?.body val subject = barcode.email?.subject } }
FirebaseVisionBarcode.GeoPoint (TYPE_GEO)
- getLat() — Returns the latitude for the corresponding geo point
- getLng() — Returns the longitude for the corresponding geo point
when (valueType) { FirebaseVisionBarcode.TYPE_GEO -> { val lat = barcode.geoPoint?.lat val lng = barcode.geoPoint?.lng } }
FirebaseVisionBarcode.Phone (TYPE_PHONE)
- getNumber() — Returns the number for the corresponding phone instance
- getType() — Returns the type of phone (FAX, HOME, MOBILE, WORK, UNKNOWN)
when (valueType) { FirebaseVisionBarcode.TYPE_PHONE -> { val number = barcode.phone?.number val type = barcode.phone?.type } }
FirebaseVisionBarcode.Sms (TYPE_SMS)
- getMessage() — Returns the message for the given SMS
- getPhoneNumber() — Returns the phone number associated with the given SMS
when (valueType) { FirebaseVisionBarcode.TYPE_SMS -> { val message = barcode.sms?.message val number = barcode.sms?.phoneNumber } }
FirebaseVisionBarcode.UrlBookmark (TYPE_URL)
- getTitle() — Returns the title for the given bookmark
- getUrl() — Returns the URL for the given bookmark
when (valueType) { FirebaseVisionBarcode.TYPE_URL -> { val title = barcode.url?.title val url = barcode.url?.url } }
FirebaseVisionBarcode.Wifi (TYPE_WIFI)
- getEncryptionType() — Returns the encryption type (OPEN, WEP or WPA) for the given network
- getPassword() — Returns the password for the given WIFI network
- getSsid() — Returns the SSID for the given WIFI network
when (valueType) { FirebaseVisionBarcode.TYPE_WIFI -> { val ssid = barcode.wifi?.ssid val password = barcode.wifi?.password val type = barcode.wifi?.encryptionType } }
Other than the data retrieved from the barcode in relation to the barcode format, as previously mentioned we also have access things such as the barcode bounds and corners. In the image below here, I’ve taken the boundingBox property of the FirebaseVisionBarcode instance that was returned by the scanning request. Using the properties of the bounding box I’ve then been able to draw these bounds onto the image of the barcode which I originally passed into the scanner. You could use these bounds to do a similar thing in your applications — doing so allows your user to know that the barcode that they are trying to scan has been recognised successfully by the scanning operation that is currently taking place in your app.
Sometimes though, you may not want to show an entire bounding box around the whole of the barcode — in situations when this is the case the FirebaseVisionBarcode also returns us a cornerPoints property. In this case you can simply use these corner points to paint the points onto the canvas. You could also use this to draw some form of custom outline on the barcode that maybe only extends to a certain point along the barcode.
Wow, Barcode Scanning with MLKit is pretty neat isn’t it! From this article I hope you have been able to see just how much simpler Firebase have made this scanning process for our applications. What are your thoughts? If you have any comments or questions, please do reach out 🙂