In 2017 I created an Android Things project called BrailleBox — a Braille News Reader for the partially sighted.
I’m really passionate about helping others and using the knowledge I have to do so, even if it means learning new things to help achieve that goal. This was the main driver for creating Braille Box, but I felt that I could go further and create something that could help others engage in areas that I myself find joy, that may not currently be so accessible (whether accessible at all, or just not quite affordable right now). The result of this project was the Chord Assist guitar — a guitar that will help to improve the experience for the mute, deaf and blind when it comes to learning the guitar.
Something that I’ve always loved is playing music. I started playing guitar at age 10 and then picked up the bass at 14 — since then I’ve played in numerous bands and a bunch of concerts/festivals. I played quite a lot of music during my teenage and education years, probably too much to be honest… but it was a big part of growing up for me. Music was both a great creative and mental outlet — it didn’t only contribute to building on my creative mindset, but it was also a great release. During stressful or down times, playing music did a great job at taking my mind off of things and just letting go for a short time, entering my own little world.
But I did that because I could. I could pick up a guitar and start learning it, whether it was reading chords or tablature from a book to watching videos to tell me how to play certain things, or even travelling to my guitar tutor. Learning the guitar was a pretty frictionless process because I didn’t have anything that may cause a hurdle in my learning process.
It is estimated that there are at ~285 million people in the world who are partially sighted. For people who have disabling hearing loss we are looking at around ~466 million. And when it comes to people are are deaf-blind, I struggled to find worldwide statistics. I found that there are ~390,000 people in the UK, which is 0.6% of the UK population — worldwide that would likely be even bigger.
When we look at these numbers together, they’re pretty high. Now, if I was partially sighted, had disabling hearing loss or even had both — would I have been able to grab a guitar and learn how to play it in the same way? Would I have been able to read that tablature? Would I have been able to watch and listen to those tutorial videos? And would I have been able to get to my guitar tutor and listen to his guidance? Who knows, maybe I would have been able to find a way to make it work for me. But it might be fair to say that there is a chance I may have missed out on some or all aspects of this. This in turn means I would have missed out on a pretty big and influential part of my life, not have been able to invest in my creative mindset via this route and also not used this hobby as an outlet for when I needed some time out.
As I think of this, it’s pretty crappy. Why should these things not be so easily accessible to everyone? Playing music brought so much to my life and I hate the thought of others missing out on such a thing because of one of these conditions. Whilst it may currently be slightly accessible, how can we make this better and explore a new way of learning music using all this great and open technology that we have?
And this is exactly where the idea came from for the Chord Assist guitar. Chord Assist was already an Actions on Google project that I built for the Google home, now I wanted to take that and stick it in a guitar powered by voice, visuals and braille. All three of these together will hopefully help to reduce the friction that may be experienced throughout the process of learning an instrument.
At first I didn’t know where I would start — but several things inspired me to push forward with this idea. To begin with, seeing this condition in my family and what it could take away from someone was a really close to home experience — whilst I was aware of these disabilities and wanting to help people, seeing the effects that close to you make things even more apparent. Watching Sundar Pichai talk about morse code technology at #io18, I literally had tingles seeing how much difference had been made to someones life — and the passion behind that. I’ve never really built a big hard ware project, so my confidence there was lacking — but having been watching Simone Giertz on youtube more recently, I had discovered a greater confidence in what I could do.
So here it is, the result. Remember to read check out chordassist.com and the attached youtube video above. This article is a little more in depth, so feel free to read on if you want to know more 🙂
I am super excited to share my side-project from the past 6 months: Chord Assist – an accessible smart guitar for the blind, deaf and mute 🎸 Powered by @ActionsOnGoogle, @Dialogflow, @Firebase, @googlecloud & @Raspberry_Pi 🙂 Read more at 👉 https://t.co/t5SoqaPdqi #a11y pic.twitter.com/lJk1pySSSh
— Joe Birch (@hitherejoe) February 11, 2019
What does this guitar do?
The main aim of this project was to bring the joy of learning music to all, I’m hoping this section will show you how I tried to achieve that:
Request how to learn a chord
Using the microphone on the guitar, the user can request to learn how to play a chord — if the desired chord is supported then the information will be returned and presented back to the user. For prototype purposes a button needs to be pressed to trigger the listening process, but a switch could simply be used to enabled and disabled the microphone as desired.
Select how to learn a chord
Using the Left, Right and Select buttons supported by an LED segment display on the guitar, the user can request to learn how to play a chord — and if the desired chord is supported, then the information will be returned and presented back to the user. This was offered as an alternative to the microphone incase the user is unable to speak — maybe they have a disability that prevents them from doing so, or maybe they are in a place that makes it difficult for them to do so.
Display the requested chord on screen
Once we are returned the information on how to play the chosen chord, if the screen is turned on then this will be used to display the chord representation in a visual format. This piece of functionality allows for users to who may want a visual representation of the chord; be it for a preferred learning method or not being able to hear the chord spoken by the speaker.
Display the requested chord in Braille
Once we are returned the information on how to play the chosen chord, if the braille reader is enabled then this will be used to read the chord representation to the user in a braille format. This allows users who may be blind to be able to read the chord that has been returned to the guitar.
Repeat the previously request chord
In some cases the user may want to request for the previous chord to be shown, spoke or read to them again. In these situations, the user can either:
- Use the microphone to request the chord to be spoken again
- Use the button on the guitar to repeat the previous chord
When either of the above is carried out, the chord will be presentation through the guitar using the functionality that is enabled at the time of the request.
Enable / Disable the speaker, screen or braille reader
On the guitar there are also 3 switches, each can be used to disable any of the output methods on the guitar.
- Disable the braille reader from displaying the selected chord
- Turn off the display
- Disable the speaker from outputting any audio content
Each of the switches contains a LED which is lit up when the switch is enabled, making it to easier to signify when a given output type is enabled.
Show loading states for all usage types
There was also some thought put into the ways in which the different output types are signified as loading. When the device is in a loading state:
- When the braille output is enabled the vibrating motor will vibrate to let the user know that the device is currently in a loading state
- When the speaker is enabled, there will be a progress sound output to let the user know that the device is currently in a loading state
- When the screen is turned on, there will be a progress spinner displayed on the screen to let the user know that the device is currently in a loading state
Operational Flow
Within the guitar there is a lot of functionality, a lot of that complexity is hidden within the body of the guitar and the logic of the software itself. There are many different moving parts that come together for the different pieces of functionality. Currently the operational flow based on user interactions looks like so:
Technologies
Now, as we can see already there are a lot of moving parts when it comes to building a smart guitar. Whilst most of our logic is going to come from Actions on Google itself (as that’s the part that houses our “smart” model), we need a way to communicate to and from that which isn’t a super-simple process.
Actions on Google SDK / DialogFlow
These technologies are used to handle the conversational aspect of the device. There are several different intents which can be launched — the main one being the ability to request how to play a chord. These SDKs are simply used to communicate with the conversational backend hosted within these cloud services. Because of this, the device requires a network connection.
Firebase Storage
When the screen is turned on and a chord is loaded, an image of the chord will be requested from Firebase Storage. Once fetched this chord is displayed on the screen to the user.
Firebase Remote Config
On the guitar the user is able to select from a collection of chords using the 4-digit segment display. So that the content of this display is able to be dynamic I made use of Firebase Remote Config to allow the chords to be changed on the fly and expanded on if I ever decided to build on this part of the functionality.
Google Cloud Text-To-Speech
Because we’re using actions on google / dialogflow to handle the conversational logic of the guitar, we have access to the textual response within the data returned to the guitar. However, we need to take this and convert it to an audio output format — this is where the Google Cloud Text-To-Speech service comes in handy.
Components
As well as these technologies there are also many components that come together to allow the guitar to be capable of serving all these different needs. Altogether the guitar cost around £260-£300 to build — I broke a few components in the process but was able to reuse some things from previous projects to help me implement some of the functionality.
Raspberry Pi (£39.99)
This is the core part of the guitar — all of the software runs on the raspberry pi. This allowed me to keep the ‘brain’ of the product small enough to fit in the guitar and also give me access to numerous components that could operate on low power. Even though this computer is inside of the guitar, i’ve set it up for remote access incase any debugging needs to take place.
Guitar (used, £40)
If we’re going to hack on a guitar, we’re going to need a guitar! This was a worrying part for me as I didn’t want to mess up the guitar too much during the woodwork process. Because if I did, it wouldn’t be cheap to replace. I picked up a used guitar locally that was good enough for this experiment.
Dremel (£19.99)
There were a lot of little holes that needed to be drilled into the guitar, as well as some rectangular shapes for the displays that were being fitted. I was made aware of a Dremel drill and this seemed like the perfect solution — if you’re ever working on something similar then it’s worth checking these out!
Switches (£3.99 for 5)
These are just some simple 5v switches to control the on and off state of things. I did some hunting specific for switches that had a ‘feelable’ notch in the on position that also lit up .
If the user is blind then they will be able to feel the on position of the switch and be aware of its state. For all people who have some ability of sight, the light will also be a quick way for them to know if something is turned on or off, without the need to touch it discover this. I also made sure that the switches I bought were of a light that was a pleasant color, I feel green to be positive and not too bright, a better choice than a negative color such as red.
5v solenoids (£5 each)
Solenoids are a component that provide a rod that pushes in / out when it receives power. I used the same component when creating braillebox — the solenoids allow me to take some word, convert it to a braille representation (embossed or not) and set the pin to be HIGH or LOW based off of the braille sequence that needs to be displayed.
5 inch LCD display (£19.99)
Because we want to show a visual representation of the chosen chord to our user, we’re going to need some form of a display. I went for a simple LCD display with HDMI connectivity so that it can be easily connected to a raspberry pi. The display also needs to be powered via a micro-usb connector — there are a bunch of other components that also need power so this is no extra work here.
4 Digit segment display (£9.99)
We want to be able to show a selection of chords on screen, allowing the user to navigate through the chords and select one which they want to know how to play. The reason for using a pHat board for this is because it allows us to only use 2 IC inputs instead of several GPIO pins.
Push Buttons (£3.99)
Push buttons are a simple component, they’re buttons that do not latch to a state when pressed. These were used to trigger the conversational action and also navigate through the available chords on the segment display. In these cases we don’t need the buttons to latch, so a push button is an appropriate solution here.
5v 8 channel relay (£7.99)
We need a way to power on/off the solenoids within our circuit — the only trouble is that the raspberry pi GPIO pins can only power at 3.3v, and 5v is required by the solenoids. With this 5v relay board, 3.3v is enough to trigger the relays on and off whilst also allowing us to provide 5v to the solenoids.
Microphone (£6.49)
I picked up a USB microphone so that I could make use of the build-in audio recording capabilities of the pi, rather than having to add another component to the circuit. All you have to do here is tweak some of the pi configuration to make it the default input method and you’re away!
Speaker and grill (£9.99)
Nothing special was needed for the speaker —just need something that can connect to the headphone jack of the raspberry pi. It could probably do with an amplifier to make it a little louder, but I plan on adding one when I iterate on some of the work here.
Portable battery pack (£23.99)
I didn’t want to have to plug in the guitar every time I wanted to use it — so for this reason I picked up a portable charger that is capable of supplying 5v power. The one I got holds a lot of charge and is able to power the pi for quite some time. I have a lose USB cable that can come out of the sound hole of the guitar for now, this could probably be improved in future though 🙂
Project boxes (£9.99 for 5)
There was a lot of wiring for each functionality of the guitar, and it was going to be all put in a pretty small place (inside the guitar). For this reason I picked up some project boxes so that each breadboard would be out of harms way from other wires and the whole thing was kept nice and tidy inside — this would make future maintenance much easier!
F-clamp and grip clamps (£20)
When it came to fixing the solenoids for the braille reader inside of the guitar, I needed something to hold them in place. I created 2 columns of 3 solenoids — I used the grip clamps here to hold each of the solenoids in place to their column whilst the wood glue dried. Followed by the F-clamp to hold
Misc (~£30)
There were a bunch of other things I used as well which don’t really need to be listed above, but if you’re interested:
- Breadboards (for prototyping)
- Solderable breadboards (for finalising the work after prototyping)
- Resistors
- Single core cable
- Wood glue
- Super glue
- Wood (for fixing things inside the guitar)
Tying the components together
Each of these components came individually, this means they needed to be wired up and connected so that communication could happen between them and the raspberry pi. Altogether these connections look something like this:
I understand that looks like a lot to take in, so let’s break each one down quickly:
LCD HDMI screen
The LCD screen used in the project doesn’t exactly look like this one, but it’s the closest I could get in the prototyping software. My screen has an HDMI connect, a ground pin and a power pin.
The HDMI connects straight to the HDMI output of the raspberry pi. The ground needs to be connected to a ground pin, followed by the power pin connecting to one of the 3.3v output pins on the raspberry pi.
4-digit segment display
In order to display the available chords for selection, a 4-digit segment display is used. I actually picked up the Four Letter pHAT from pimoroni for this. It uses a GPIO ribbon cable to connect to the raspberry pi, but only a few of those connections are actually used. It makes use of I2C to handle the communication to and from the pi — this meant that no additional GPIO pins were required to set the content of this segment display.
Solenoids
The solenoids are used to drive the braille display. The GPIO pins of the pi only provide 3.3v of power, so these aren’t enough to trigger the 5v solenoids. To get around this I made use of 5v relay channels — this can be triggered by 3.3v and are capable of supplying 5v power (as the relays themselves are powered by 5v). Whilst the diagram looks pretty complicated, it actually only consists of:
- Each solenoid needs to be connected to a ground and the power output of the relay channel
- Each relay needs to have its channel input connect to a power source (on top, next to where the solenoid is connected), this will in turn power the solenoid when the switch is enabled
- Each relay needs to be connected to a ground source, a power source and also a GPIO pin from the raspberry pi. This GPIO connection will then be used to enable and disable the relay channel.
Push buttons
Each push button has a ground and power connection. For the power, this is connected directly to the power source from the raspberry pi. The ground connection then needs to be routed through the GPIO trigger cable and back to the ground source of the raspberry pi. When this button is pushed, this connection to the ground will send a signal to the connected GPIO pin which can then be handle in the software.
Toggle switches
The toggle switches work in a similar way to the push buttons, their behaviour is slightly different though. Each switch needs to be connected directly to the power source of the raspberry pi, as well as having a GPIO pin connected directly to one of the connection points.
For the ground connection of the switch, this needs to be connected to the ground point of the breadboard through a resistor. When the state of the switch is changed, a signal will be sent to the connected GPIO pin.
Coding the software
You might be curious about how the code looks for this project, so I’ll share a few bits here. I won’t outline the entire software as a lot of the things are quite similar in how they work, but hopefully the below gives an idea of how things are built.
Braille display
The braille format that has been used consists of 6 units, any textual content needs to be converted into this format so that it can be displayed on the braille pins. For this, we first need to begin by defining the alphabet and any symbols against their given braille representation:
https://gist.github.com/hitherejoe/4cbf1c1eaeaf362a4a032d1098c9988c
Now that we have braille representations defined for each of the above, we can take our given text and build a collection of braille sequences that need to be printed out. The function below takes our actions on google textual response, loops through each of the characters and then adds the braille representation for that character to our result array.
https://gist.github.com/hitherejoe/3d2b2ef03a5eb54bc396f52cb3140971
Now that we have the braille representations, we need to display them using the solenoids. Because our solenoids are connected via GPIO pins, we just need to take each of the 6 units in each braille sequence and set the corresponding solenoid to be either HIGH or LOW based on that value. You can see in the function below that we loop through each of the braille sequences, display the braille sequence on the pins, sleep for a short period and then repeat for each sequence in the collection.
https://gist.github.com/hitherejoe/d0b4b5dfa4daa2c5f3e4c5852a577469
4-digit segment display + Navigation
For the display navigation we simply make use of the RPI.GPIO library to allow us to hook into GPIO communication. Here we set a listener for each of the navigation buttons, linking to an event handler for one of them is triggered.
https://gist.github.com/hitherejoe/e38f48252c274d3b30113f85985e8e6e
For the display, we make use of the library provided for the four letter phat which allows us to easily display textual sequences on segment display.
https://gist.github.com/hitherejoe/732bd97286535b1a9daf64ad557f7796
And when each of the navigation buttons is pressed, we simply set the position of the selected chord and then call the display_chord function so that our segment display is updated.
https://gist.github.com/hitherejoe/d337d1a6d4f444a0691dab5508e61507
When the submit button is pressed, we simply trigger the intent text detection function. This triggers a dialogflow request using the selected chord as the argument for the sentence.
https://gist.github.com/hitherejoe/ec78627198ca3f6589c6da4fc0777653
Actions on Google text query
We perform textual queries using the dialogflow client library for python. All we need to provide here is the project ID, a session ID (this is used for continuing conversations within the current context), the desired language code and the input text used for the conversation. This is a blocking request and the responses is returned from the client once the request has been completed.
Once we have this response we can access it’s data for use however we desire. You can see here that I access the query_result, along with the fulfilment_text which gives us the textual response provided by dialogflow.
https://gist.github.com/hitherejoe/7249bbfd4ff74687dff47762a883c2ff
Actions on Google voice query
The voice query works a little bit different from the above, however, we are able to reuse the code from the text fulfilment function! Here we make use of the Google Cloud CloudSpeechClient class from the Text-to-Speech functionality of Google Cloud. All we do here is start listening for vocal input (once the button has been pressed to trigger it) and then once text has been detected, we pass the parsed audio content in a textual format to the detect_intent_texts function.
https://gist.github.com/hitherejoe/463c43c02332e38fa5d13728ddf5b09a
Displaying the chord image
Within the chord document from firebase we get the details which are required for displaying the chord image. We first need to fetch that image from Google Cloud storage. Here I make use of the library which handles these requests for me — I just had to provide the required ID and information on the file that was to be retrieved. This image could then be used for display inside of the UI component for the screen.
https://gist.github.com/hitherejoe/f40d39d66d6568a044f5f943bd0a2903
Enable / Disable features
The enabling and disabling of output modes works in a simple way. I keep a collection of three boolean flags which state wether or not an output mode is enabled — if the out mode is not enabled then when a response is received, that output mode is ignored. For each switch, a GPIO listener is set for when the switch is interacted with. When this happens a toggle function is called which is used to set the value of the corresponding boolean flag.
https://gist.github.com/hitherejoe/fd160058ec0e2eef4f595952ec4f73e6
To conclude
This was a super fun project to build — when you mix some of your biggest passions you know that you’re going to build something meaningful and have a great time doing so. I really hope that in the near future we will see more and more ideas and technology like this in the market. Music is an area that hasn’t been explored too much during my research, and as I previously stated — music has been a huge part of my life, everyone should be able to experience it.
This project is currently a proof-of-concept, or prototype. It’s likely that i’ll build on this and improve some things on the guitar itself. I also plan on releasing the Actions on Google app to the app store so that it can be used via Google Home devices. In the meantime though, I’d love to hear any thoughts / feedback or general comments on this project 🙂