Score Viewer 2.0 for Glass

edward

Andre, Cindi, and I all reading music notation from Glass.

Last November, we finished the first version of the music score viewing application. Built on the Mirror API, the app pushed a series of cards (what we had as the score of Armenian Dances by Aram Khachaturian) to Glass. Although functional, the app had a few problems.

  1. Google App Engine (GAE) would occasionally timeout while sending cards to Glass. I didn’t solve this issue fully, but from what I could tell, since we were sending many calls to Glass, the program would stall waiting for a response and terminate. This was far from ideal, and even at GAE’s highest timeout threshold it would still stall on slow applications. A possible fix for this would be to have a different thread responsible for sending the cards, and a loading screen telling the user to wait. Since I knew we were going to start from scratch on this next version, I never implemented this.
  2. The original app sent all different cards, versus stacked in one bundle. If the user received an email or text message while the bundle of score pages was being pushed to his or her Glass, that would be inserted in the middle of the score. Awkward. I was able to fix this by bundling the cards in one stack.
  3. If you received new cards while performing, they would take precedence over your score. Turning off wifi and Bluetooth fixed this, but that’s not truly an ideal solution.
  4. Controlling the cards was limited to the gestures on the touchpad or through a Bluetooth controller. We used a wireless keyboard to advance pages, but pairing the device was awkward and required a hack to work.

For these reasons, I immediately started using the Glass Development Kit (GDK) to write the next version of the score viewer. Setting up the programming environment for Glass is pretty straightforward and is discussed in the documentation for the GDK.

At its core, the application is very straightforward. An ImageView loads up the first page of a score, which is simply a 640x360px image. After a certain event occurs, the next image is loaded. After all are cycled through, the score returns to the first page.

Here is the cool feature: to advance pages, the user blinks twice while looking at the Glass display. To go to a previous page, the user taps the touchpad once. This decision was a conscious one; tapping the touchpad is faster than swiping back and forth. In order to get this functionality to work, I used Takahiro “Poly” Horikawa’s Eye Gesture Library on Github, available here.

So, how well does it work? Truthfully, quite well. After getting used to looking through or away from Glass, I’m always successful in having the pages advance when they should. Occasionally a “phantom blink” happens, and the page advances before it should. A quick tap brings it back in place. This could be disastrous in live performance, especially if musicians are reading notation off of the device, versus having a conductor use it as a reminder of meter changes and cues. I’ve found that calibrating the winking sensor can greatly reduce phantom blinks. Also, be sure the “wink for photo” setting is enabled in the settings menu before using the app.

Although a bit messy, the source for the Glass Score Viewer 2.0 is available on Github below. The demo score installed is Khachaturian’s Armenian Dances. I hope to comment some more of the code and upload a revised version soon.

https://github.com/tehrlich/gdkscoreviewer

Camera + Image Transfer Tool

Now that the metronome is underway, it’s time to get started on the still image transfer tool. The concept is to write a script (or find/write an app) to constantly take photos on Glass, and then upload them to a sharing service (likely Dropbox.) I decided to start on the second task first, since it’s more difficult. Unfortunately, Dropbox has not yet written a daemon for Glass. A few months ago, Barb Friedman sent Zapier my way. This looks perfect, but as of now, Zapier can only respond to three actions on Glass:

None of these satisfy our needs, since they require user interaction. I started to look around and found a GitHub project that allows users to share photos to Dropbox via Glass. I was just about to spin up a server to install Node.js, until I realized that it only adds another “Share” option; it doesn’t work as a daemon constantly watching a folder to update.

So what’s the plan?

I’m hoping to download the Dropbox APK, authenticate using a borrowed Bluetooth keyboard from Academic Technologies, and go on from there. Although there is an easy way to screencast to my phone, there isn’t an easy way to do text input. I did find this Google+ post to pair a Bluetooth keyboard with Glass.

More info soon!

Sweeney Todd Through Google Glass

Similar to Cindi’s recording with the Syracuse Society for New Music, I wanted to try recording my conducting using Glass. This video is from a very early tech rehearsal, and I posted it for the sheer number of demonstrative qualities that a Glass recording reveals. (As in, the errors you can hear are now fixed!)

First check out the video:

Here’s what you’re seeing:

0:07: I quietly breathe to have the players enter together, but it sounds quite loud on Glass.
0:23: I gesture and whisper to the pianist to continue holding that organ chord.
0:39: I lean over to the Assistant Pit Director to make a rehearsal note: Add a fermata song #4 measure 5.
0:49: The percussion keyboard should be on bells, not xylophone.
0:53: The violinist needed to change a string, so I quietly sing his part to myself. Doesn’t sound quiet here!
1:00: I look over the cellist to fix the intonation.
1:25: I ask the bassoonist to play that initial quarter note much stronger.
2:08: The tempo is initially too slow. To indicate the need to move faster, I begin to move my head up and down around 2:10.
2:26: Big cue to the strong trumpet and horn entrance.
2:40: I quietly say “ow” to myself, as a reminder to fix the intonation in the waltz section.
3:02: Big cue to the stopped horn.
3:15: Eye contact with the percussionist for the accelerando.
3:35: Big cue to the trumpet solo.

My thoughts?

Ouch!: After about 15 minutes, I really began to feel how hot Glass was on my head.
Volume: As Cindi addressed in her own post, all of my comments, singing, and breathing are really loud.
Lighting: Granted we are sitting in darkness, but the low light quality isn’t the best. The bright light facing me is so the actor’s can see me via a television hung above the audience.
Revolutionary: After watching this video, I immediately knew the spots I needed to address in rehearsal. You’re likely thinking to yourself, “just use a video camera!” I do this already; I record all of my rehearsals with CU Winds. The problem is that the audio from the back of the room is all percussion. I need have two devices, an audio recorder in the front of the room, and a camera in the back of the room, and then merge them. While I’m not getting the perspective of watching me conduct, I’m still getting the perspective of seeing the players. It still shows me how much time I spend looking at the score and the amount of extraneous movement I have. Both are critical in assessing my own conducting.

Although this is still a bit of effort, if I was wearing Glass constantly, it wouldn’t be. I wouldn’t need to set up any recording equipment, charge batteries, transfer files over USB (Glass automatically does this.) It would be a seamless way to audit rehearsals after the fact. As someone who is constantly critiquing his own conducting and rehearsal technique, I find this essential.

BeatKeeper Version 1.0

Cindi and I spent a few hours two weeks ago getting BeatKeeper up and running on Glass. After cloning the repository on my local machine, I imported the app into Eclipse and pushed it to the device. We were immediately presented with something that looks like this:

It was quite exciting for us to see a metronome! I swiped forward and back to navigate through the controls; I hit play and a consistent pulse started in my ear. Perfect.

Next thing was making it landscape and more easy to read. I headed into the AndroidManifest.xml file and changed android:screenOrientation="portrait" to android:screenOrientation="landscape". Now we were presented with something a bit more usable, but the Start button was cut off:

This was a great time for me to introduce layout files to Cindi, so I went over the XML and explained how the placement worked. We messed around with the layout and sizing and ended up here:

After our NYC media lunch, I put in a lot of thought about usability. Buttons are appropriate for mobile devices, but not really for Glass. Using the slider and tap gestures are considerably more intuitive and simple. In their usability guidelines, Google writes, “Glass users expect the technology to be there when they want it and out of the way when they don’t. Your Glassware should never take precedence over users’ lives.” Especially in a rehearsal setting, I wanted to design something simple and intuitive.

Here is the first real version of BeatKeeper for Glass:

It’s simple and easy to read. To adjust the tempo, simply slide your finger on the side of Glass. To start and start the metronome, tap once. All of the gestures are recognized via a GestureDetector. I’ve stripped all unnecessary controls from the app. The resulting experience is sleek and straight forward to use.

So what’s next for BeatKeeper for Glass?

A reverse metronome is a tool for finding a tempo via tapping. Over the next few days, I hope to create another simple tool that averages taps on the device. I’m not sure if it will be a standalone application or integrated with BeatKeeper. Although the two go hand-in-hand, it would be difficult to switch functions between a metronome and a reverse metronome. A long press on the side could change the mode of the application, but if you press and hold for around two seconds, you are presented with a Google Search. There is an intermediary gesture of around one second, onShowPress, but if you hold it down for too long, you’re presented with the search window. The only way to override this, as far as I know, would be to root the device and edit the Glass interface itself. The other idea is to utilize swiping up and down to change the functionality.

The (somewhat hastily written) source is viewable on Github here:

https://github.com/tehrlich/BeatKeeper

The Sexiness of Mystery and News Ways of Thinking (and Performing?)

You probably know that Google contacted all the Explorers and invited them to invite 3 more explorers. Another good marketing campaign but one wonders if they are still not sure the thing will sell on the open market. (We had to make sure that our 3 chosen ones could cough up the $1500.) By keeping it a mystery and open to only “the chosen,” they keep it sexy, right? And speaking of mystery:

http://www.allvoices.com/contributed-news/15875665-mystery-barges-unmasked-as-luxury-showrooms-for-google-glass-devices

You can bet that Google orchestrated that entire media hype surrounding “the mystery” of the barges.

ANYWAY, I chose 3 explorers based on a new idea. I can admit it now…I love technology. I think it’s exciting and it’s helping us evolve. (Other than getting taller as a species, what else is happening?) Technology changes how we think. It changes our BRAINS. Your brain has changed because of the internet.

http://www.amazon.com/The-Shallows-Internet-Doing-Brains/dp/0393339750

http://www.huffingtonpost.com/shaahin-cheyene/internet-brain_b_1248845.html

http://www.onlinecollege.org/15-big-ways-the-internet-is-changing-our-brain

Just a few examples…there are MANY more. I actually like this. I think it is interesting and stimulating how technology encourages us to think in new ways. (Granted, some of these things seem scary…like the concept of not bothering to memorize or not giving anything our FULL attention.) It’s why we started to think about notating music in a new way. Our first thought was to embed a traditional, condensed score on Glass and perform from that. (We are still thinking about doing this.) But, Glass has inspired a new way to think. The screen is so small! And we’ve been frustrated about how to manipulate the pdf of our score. But these limitations have inspired something else. Why use a traditional score at all? This is how we developed the concept of a “salient” score. By the time we get to the performance (or, if we’ve really done our homework, the first rehearsal), we know the score. We don’t need ALL the information–just reminders. A salient score.

Back to the three chosen ones. Kevin Ernste (colleague and composer with a specialization in digital musics), Tyler (co-researcher and clarinet player), and Andre Hafner (information technology specialist in the College of Architecture, Art, and Planning, keyboard player and killin’ vocalist in pop and rock musics).

Here’s the new idea…why not compose a piece specifically FOR Glass, which we all perform (I’ll do percussion and laptop) reading from Glass. Stay tuned!!

A Metronome and A Salient Score…Plus a Little Media Attention.

We’ve been busy at work since our Glass demonstration in Bailey Hall on Sept. 29th. In addition to Tyler successfully pushing a pdf to Glass, he and I spent a few hours in my office the other day and we were able to use some existing open source Android code from a metronome app called “Beat Keeper.” The transfer of it to Glass posed some problems (and I am learning that the majority of programming time is spent searching for error codes on the internet and finding solutions that other people have posted) mostly because Beat Keeper was designed for a smart phone. Think about the differences between a smart phone and Glass–smaller screen, landscape versus portrait orientation, lots of buttons to navigate versus no buttons to navigate, the sound, etc. However, with a few touch ups here and there, Tyler was able to at least embed the metronome and we were able to navigate through it and get it to work. Tyler will show you a screen shot of the work so far when he writes his blog. Now, we have to come up with a way to simplify it, a way for it to work through voice commands, and a way for it to easily shut on and off. But, it’s a big step forward and we are encouraged to keep going.

The other step forward has been in our thinking. Given the size of the Glass screen and how complicated (but not impossible) it is to manipulate pdfs on Glass, we are starting to re-think about how we want the score to look. I’ve always been fascinated by the concept of freedom within discipline and Glass’ limitations have spurned us to thinking differently about the notation of music. Here’s what I mean…by the time we conductors get to a performance, our scores are pretty much memorized. If not memorized, we know them VERY well. So, what if what was embedded on Glass wasn’t the full score (or even a condensed one), but something we are calling a “salient score.” Just those reminders that we need as a piece progresses…what instruments are playing at any given moment, important cues and dynamics, changes in tempo or time signature. We don’t even need the full time signature…just “4″ or “1″ or “quarter = eighth” etc. And because we know the score so well, we probably don’t need the melody or harmony…just the shape and harmonic changes. Perhaps, depending on the piece, a shape or directional line? Perhaps colors for dynamics? You get the idea. We are thinking of ideas now and welcome any that you may have as well.

In terms of live streaming a piece through Glass, you already know the big problem: latency. So, we have come up with a solution. Glass is great at taking pictures. It’s fast and the quality is terrific. What if Glass took a series of still images as the piece was progressing…say, one every 2-5 seconds or so. And those images were projected. We might try both…one piece live streamed with the lag and one using the still images. Stay tuned!

One of the things that has taken me by surprise is the amount of attention that Glass has received in the media. Last Friday, Tyler and I were invited to New York City for a media luncheon. Journalists from the LA Times, Wall Street Journal, The Verge (this one is a very big deal…I didn’t know that until after the conference), International Business Times, and other magazines and online journals from as far away as China and Spain attended. Tyler and I gave our talk and demonstrated our research so far and they asked very good questions. Two articles have already been published:

International Business Times:
http://www.ibtimes.com/google-glass-concert-hall-cornell-prof-looks-make-beautiful-music-high-tech-headgear-1442290

The Verge:
http://www.theverge.com/2013/10/27/5034818/cornell-professor-cynthia-turner-mixes-google-glass-and-classical-music

These two journalists did a great job of summarizing our research so they are worth a read. Also, if you missed the You Tube video I posted of me wearing Glass during a new music concert, it is in the Verge article. Apparently there are more articles coming. Here are three things I’ve learned by this ‘media’ experience:

1. Do NOT read the comments sections of online articles. Ouch. Travis Cross reminded me of this when he sent me a clip of CJ Cregg ranting to Josh Lyman. (see FB post)
2. There are a lot of naysayers out there.
3. Be VERY careful what you say to journalists. It might be a trap. One of the journalists asked me what my “pie in the sky” vision for Glass was. I should have said, “I don’t know.”
4. Cornell has a big presence in the city…bigger than I thought.

Before our trip to NYC, I had a great visit from Demian Caponi from Google. He was on campus recruiting and stopped in to meet me. What an awesome guy. After we chatted for a while, I asked him what he did at Google besides recruiting and he told me that he taught a class on “Presenting with Charisma.” Really? Tell me about it! And everything that came out of his mouth was pretty much the same things we talk about in conducting class…presence, body awareness, leading with the heart, voice awareness, gesture, posture, stance, eye contact, tone, etc, etc. I invited him to come and talk to my class and he was great. Here’s a picture of he and I after class:

So, onward we go with our research and experiments. Thanks for tuning in.

Demonstration of Google Glass in Bailey Hall

On Sunday, Sept. 29, CU Winds performed the first concert of the season. Music by Michael Gandolfi, H. Owen Reed, Eric Whitacre and Cindy McTee. Tyler and I gave a talk on our research with Google Glass and undertook an audience poll on whether Glass live video-streaming from the conductor’s perspective who enhance or detract from the concert experience. We also fielded questions from audience members, some of which only Google can answer! See the concert and presentation here:

http://bailey.webcast.video.cornell.edu/Mediasite/Play/3719c16f0ed5455092a420367c7d49e21d

A video performance #throughGlass

I conduct a professional new music ensemble in Syracuse (societyfornewmusic.org) and we had an interesting performance last night at the Everson Museum. Greg Wanamaker composed a haunting work called A Story Within a Story which we performed last year along with Carrie Mae Weems (carriemaeweems.net) video installation. It was a highly successful performance so Greg composed some addition music for it and we added dance segments along with the video. Our dancer last night was Aisha Mitchel, principal dancer from Alvin Ailey. (Being backstage with her and watching her warm up was an incredible experience…unbelievably beautiful to watch.) It is a wonderfully collaborative work powerfully addressing the poetics of identity and historical memory. This was the opening act of the Crave Festival.

I wore Glass and recorded what I could before the battery ran out. You can see the video here:

http://youtu.be/smon-nW7p9k