Adam T. Bradley

Posts in category “rstats”

PaperJS Sparklines

Yet another PaperJS experiment. This one generates Sparklines. It simply looks for canvases matching a specific CSS selector (canvas.sparkler by default), and draws a chart based on JSON-formatted data in the data-data attribute. Here, I'm displaying some unemployment data from Quandl.

The code is on Gist.



ct.unemp <- Quandl("FRBC/UNEMP_ST_CT", trim_start="1978-01-01", trim_end="2013-12-01", collapse="monthly")
ct.2yunemp <- rev(ct.unemp$Value[1:24])

me.unemp <- Quandl("FRBC/UNEMP_ST_ME", trim_start="1978-01-01", trim_end="2013-12-01", collapse="monthly")
me.2yunemp <- rev(me.unemp$Value[1:24])

ma.unemp <- Quandl("FRBC/UNEMP_ST_MA", trim_start="1978-01-01", trim_end="2013-12-01", collapse="monthly")
ma.2yunemp <- rev(ma.unemp$Value[1:24])

nh.unemp <- Quandl("FRBC/UNEMP_ST_NH", trim_start="1978-01-01", trim_end="2013-12-01", collapse="monthly")
nh.2yunemp <- rev(nh.unemp$Value[1:24])

ri.unemp <- Quandl("FRBC/UNEMP_ST_RI", trim_start="1978-01-01", trim_end="2013-12-01", collapse="monthly")
ri.2yunemp <- rev(ri.unemp$Value[1:24])

vt.unemp <- Quandl("FRBC/UNEMP_ST_VT", trim_start="1978-01-01", trim_end="2013-12-01", collapse="monthly")
vt.2yunemp <- rev(vt.unemp$Value[1:24])
New England Unemployment by State, 2011-12-31 to 2013-11-30
State Min/max 2011-12-31 2013-11-30
Connecticut 7.5%/8.5% 8.3% 7.5%
Maine 6.4%/7.4% 7.4% 6.4%
Massachusetts 6.7%/7.2% 6.9% 7.1%
New Hampshire 5.2%/5.6% 5.4% 5.2%
Rhode Island 9.4%/10.9% 10.9% 9.4%
Vermont 4.2%/5.1% 5.1% 4.3%

RMarkdown and Jekyll

While migrating my blog recently from WordPress to Jekyll, I looked for easy ways to post RMarkdown documents to my blog. I ended up developing two options of my own (available on GitHub):

jekyll.R provides two R functions to let you add posts or drafts to your Jekyll blog direct from the RStudio console, by knit()ing an .Rmarkdown document into markdown with the YAML front-matter Jekyll expects. This is probably the best option for blogs.

rmarkdown_converter.rb is a Jekyll plugin that provides an Rmarkdown converter. This lets write your posts in RMarkdown and save them in your site's _drafts or _posts directory with the extension Rmd or Rmarkdown, just like you normally post standard markdown documents.

The downside here is that changes to your R environment (you get a new computer and don't reinstall all of the packages you had before, you upgrade R and one of the packages you used to use is no longer supported, etc.) can make it impossible to build your blog (or at least some posts). It might be more useful for some non-blog sites.

This code borrows heavily from the sample converter on the Jekyll plugins documentation page and this post by Simon Elliston Ball.

More Shiny

Another Shiny app, this one taking data on U.S. unemployment from Quandl and generating graphs and tables describing it. I was mainly trying to update my Shiny knowledge based on the new(-ish) version 0.4.


This app is on GitHub, so it should be possible to run it with:

runGitHub('shiny_unemployment', 'atbradley')

What’s Matt Listening To?

Matt Asher, over on his blog, analyses the playlist from a local pop-music station in Toronto, and provides an interesting script for generating HTML-based charts in R. He wonders how Radio Paradise‘s playlist would look in comparison.

I copy-pasted five days worth of Radio Paradise’s playlist into this file and tinkered with it in R.

source_gist(<a href="" target="_self">5034250</a>)

We have here 1530 individual plays, with 1274 unique tracks. That already sounds like more variety than Toronto’s Virgin Radio offers–most songs on RP were only played once. This makes the kind of chart Matt did for Virgin Radio a little boring here:

cat(HTML.chart(smry, 250, ytLinks = T))
   Band of Horses: On My Way Back Home
   Fleet Foxes: Your Protector
   The Derek Trucks Band: This Sky
   Anna Ternheim: Bow Your Head
   Big Sugar: Come A Little Closer
   Calexico: Sinner In The Sea
   Cult With No Name: Under The Dirt
   Cult with No Name: You Know Me Better Than I Know Myself
   Death Cab for Cutie: Home Is a Fire
   Elvis Costello: Pump It Up
   Fink: Warm Shadow
   Jenn Grant: The Fighter
   Johnny Marr: The Messenger
   Keren Ann: My Name Is Trouble
   Laura Marling: Devil’s Spoke
   Neil Halstead: Full Moon Rising
   Night Beds: 22
   Night Beds: Ramona
   Radical Face: Black Eyes
   Rebecca Zapen: Colorado
   Soha: C’est bien mieux comme ca
   Stereophonics: In a Moment
   The Cinematic Orchestra: To Build A Home
   The Jam: Town Called Malice
   The Slow Show: Falling Slow
   1 Giant Leap: The Way You Dream (feat. Michael Stipe and Asha Bhosle)
   Aimee Mann: Charmer
   Air: La Femme D’Argent
   Alpha Rev: New Morning
   Alt-J: Something Good
   Alt-J: Tessellate
   Amy Winehouse: You Know I’m No Good
   Andrew Bird: Tenuousness
   Ane Brun: These Days
   Anna Ternheim: What Have I Done
   Anni B Sweet: Take On Me
   Antje Duvekot: Phoenix
   Arcade Fire: We Used To Wait
   Barra and Govinda: Through The Veil
   Basia Bulat: Go On
   Beck: Beautiful Way
   Beck: Paper Tiger
   Ben Harper: Jah Work
   Beth Orton: Dawn Chorus
   Beth Orton: Magpie
   Blood Sweat & Tears: I Love You More Than You’ll Ever Know
   Bob Dylan: Don’t Think Twice, It’s All Right
   Bob Marley: Sun Is Shining
   Bohren & Der Club Of Gore: Constant Fear
   Bonobo: El Toro
   Bonobo: Nitelight
   Bruce Cockburn: Wondering Where the Lions Are
   Built To Spill: Gone
   Built To Spill: Wherever You Go
   Cake: Love You Madly
   Calexico: Splitter
   Cat Power: Cherokee
   Choir of Young Believers: Hollow Talk
   Chris Whitley: To Joy (Revolution of the Innocents)
   Christians and Lions: Skinny Fists
   Claude Chalhoub: Don’t Wake Me Up
   Coldplay: Spies
   Cowboy Junkies: Lay it Down
   Crowded House: Private Universe
   Dandy Livingstone: Rudy a Message to You
   Danger Mouse & Daniele Luppi: Problem Queen (w/ Norah Jones)
   Dave Matthews Band: Broken Things
   David Bowie: Ziggy Stardust
   Death Cab for Cutie: Doors Unlocked and Open
   Dengue Fever: Uku
   Depeche Mode: Precious
   Dire Straits: Ride Across The River
   Dire Straits: Sultans Of Swing
   Dire Straits: The Man’s Too Strong
   Dire Straits: When It Comes To You
   Doc Watson: Windy and Warm
   Donovan: Sunshine Superman
   Doves: Lifelines
   Duke Ellington: How High The Moon
   Eastmountainsouth: The Ballad Of Young Alban and Amandy
   Elbow: Grounds For Divorce
   Elbow: Lippy Kids
   Elbow: Red
   Elephant Revival: Cosmic Pulse
   Elliott Brood: The Valley Town
   Emily Barker & the Red Clay Halo: Fields of June (w/ Frank Turner)
   Fleetwood Mac: Landslide
   George Harrison: What Is Life
   Gillian Welch: Scarlet Town
   Githead: Take Off
   Gomez: Just As Lost As You
   Gooding: First Lullabye
   Great Lake Swimmers: Your Rocky Spine
   Gregory Alan Isakov: That Moon Song
   Grinderman: Star Charmer
   Grizzly Bear: Speak In Rounds
   Ian Moore: Civil Light
   Interpol: The Scale
   Ismael Lo: Tajabone
   Jack White: Love Interruption
   Jenny Lewis: Pretty Bird
   Jesse Sykes & The Sweet Hereafter: The Dreaming Dead
   Jimi Hendrix: The Wind Cries Mary
   Jimmy Cliff: The Harder They Come
   Joan Osborne: Brokedown Palace
   John Cale & Brian Eno: Lay My Love
   John Lee Hooker: Dimples
   John Lee Hooker: This Is Hip
   Joni Mitchell: Cold Blue Steel And Sweet Fire
   José González: Stay in the Shade
   José González: Killing For Love
   Junip: Line Of Fire
   Kate Miller-Heidke: The Devil Wears a Suit
   Kathleen Edwards: Goodnight, California
   Led Zeppelin: Dazed and Confused
   Loreena McKennitt: Marco Polo
   Los Lobos: Bertha
   Lucinda Williams: Rescue
   Ludovico Einaudi: Divenire
   Ludovico Einaudi: Primavera
   Luna: California (All the Way)
   Lyle Lovett: Friend Of The Devil
   Mark Knopfler: Privateering
   Medeski, Martin & Wood: Ice
   Morcheeba: Enjoy the Ride
   Morcheeba: Over & Over
   Morcheeba: The Music That We Hear (Moog Island)
   Muddy Waters: I Can’t Be Satisfied
   Neko Case: Hold On, Hold On
   Neko Case: Things That Scare Me
   Nick Drake: Pink Moon
   Nick Drake: Things Behind The Sun
   Norah Jones: Light as a Feather
   Norah Jones: Say Goodbye
   Orchestra Baobab: Dee Moo Woor
   PJ Harvey: On Battleship Hill
   PJ Harvey: You Said Something
   Patti Smith: Nine
   Paul Weller: Dragonfly
   Peter Gabriel: Digging in the Dirt
   Peter Gabriel: Sky Blue
   Peter Gabriel: Solsbury Hill
   Peter Tosh: Mystery Babylon
   Pinback: Sender
   Queens Of The Stone Age: No One Knows
   R.E.M.: New Test Leper
   Rachel Zeffira: The Deserters
   Radiohead: Staircase
   Radiohead: Street Spirit (Fade Out)
   Ray LaMontagne: Be Here Now
   Red Rider: Lunatic Fringe
   Redd: Hala Ask Var Mi
   Rilo Kiley: Dreamworld
   Rilo Kiley: Hail To Whatever You Found In The Sunlight That Surrounds You
   Rolling Stones: Monkey Man
   Rolling Stones: Time Waits For No One
   Roxy Music: Love is the Drug
   Sade: No Ordinary Love
   Santana: Se A Cabo
   Sarah Blasko: I Awake
   Sarah McLachlan: Sweet Surrender
   Serafina Steer: Night Before Mutiny
   Shivaree: Goodnight Moon (Live at KCRW)
   Silversun Pickups: Draining
   Son Volt: Live Free
   Sophie Zelmani: Hard To Know
   Specials: Ghost Town
   Spirit of the West: And If Venice is Sinking
   Stevie Wonder: Superstition
   Stone Roses: Fools Gold
   Stone Roses: I Wanna Be Adored
   Stranglers: Golden Brown
   Sun Kil Moon: Gentle Moon
   Supergrass: Hollow Little Reign
   Supergrass: Tales Of Endurance
   Suzanne Vega: Rock in This Pocket (Song of David)
   Talking Heads: Crosseyed And Painless
   Tame Impala: Elephant
   The Bad Plus: Flim
   The Barr Brothers: Beggar In The Morning
   The Beatles: Come Together
   The Beatles: Dear Prudence
   The Black Keys: Gold On the Ceiling
   The Black Keys: Sister
   The Cure: Close To Me
   The Cure: Just Like Heaven
   The Cure: Lullaby
   The Dandy Warhols: Love Song
   The Decemberists: Calamity Song
   The Decemberists: Don’t Carry It All
   The Donnis Trio: Tip of the Tongue
   The Doors: People Are Strange
   The Kinks: A Well Respected Man
   The Lumineers: Stubborn Love
   The Police: Driven To Tears
   The Psychedelic Furs: Heaven
   The Republic Tigers: Buildings and Mountains
   The Shins: Bait And Switch
   The Shins: Black Wave
   The Spring Standards: Watch the Moon Disappear
   The Veils: The Leavers Dance
   The Wallflowers: Reboot The Mission (w/ Mick Jones)
   The Who: Love, Reign O’er Me
   Thea Gilmore: All Along The Watchtower
   Thievery Corporation: Forgotten People
   Thievery Corporation: Lebanese Blonde (w/ Pam Bricker)
   Thievery Corporation: The Richest Man in Babylon
   Tom Petty: Breakdown
   Tom Petty: You Got Lucky
   Tom Waits: Please Call Me, Baby
   Tor: Glass & Stone
   Tori Amos: Strange Little Girl
   Trace Bundy: Bristlecone
   Trentemøller: Sycamore Feeling
   U2: The Unforgettable Fire
   VAST: I Know How To Love
   Vetiver: You May Be Blue
   Vienna Teng: Blue Caravan
   Waifs: Lighthouse
   Ween: Flutes of The Chi
   Wilco: Outtasite (Outta Mind)
   Wilco: You Are My Face
   Wild Nothing: Shadow
   Wye Oak: Civilian
   Yo La Tengo: The Point of It
   Yoav: Yellowbrite Smile
   Younger Brother: Train
   thenewno2: Never Too Late
   !Deladap: Lautlos
   10,000 Maniacs: Don’t Talk
   A Filetta: Like a Breath of Air
   A Fine Frenzy: Pinesong
   A Fine Frenzy: They Can’t If You Don’t Let Them
   A Man Called Adam: Yachts
   Aaron Copland: Fanfare for the Common Man
   Afro Celt Sound System: Big Cat
   Afro Celt Sound System: Mojave
   Afro Celt Sound System: Onwards
   AfroCubism: Djelimady Rumba
   Agnes Obel: Riverside
   Aimee Mann: Calling It Quits
   Aimee Mann: Great Beyond
   Aimee Mann: Humpty Dumpty
   Aimee Mann: Wise Up
   Air: Alone In Kyoto
   Air: Mer Du Japon
   Air: Space Maker
   Alexi Murdoch: Towards The Sun
   Alison Krauss & Union Station: Lie Awake
   Alison Krauss / Gillian Welch: I’ll Fly Away

Charting the artists played rather than the tracks gives us a more interesting chart, and reveals that someone at Radio Paradise really likes the Beatles and U2:

cat(HTML.chart(smry.artists, 250))
   The Beatles
   The Shins
   Dire Straits
   Peter Gabriel
   Porcupine Tree
   Rolling Stones
   The Cure
   Elvis Costello
   The Black Keys
   Tom Petty
   Cowboy Junkies
   Los Lobos
   Talking Heads
   Beth Orton
   Bob Dylan
   Bob Marley
   Fleetwood Mac
   Neko Case
   Norah Jones
   Pearl Jam
   The Dandy Warhols
   The Decemberists
   Thievery Corporation
   Aimee Mann
   Andrew Bird
   Anna Ternheim
   Arcade Fire
   Chris Isaak
   David Bowie
   Death Cab for Cutie
   Fleet Foxes
   John Lee Hooker
   Led Zeppelin
   Nick Drake
   Night Beds
   Patty Griffin
   Pink Floyd
   Radical Face
   The Derek Trucks Band
   Tori Amos
   Band of Horses
   Ben Harper
   Big Sugar
   Built To Spill
   Crowded House
   Elliott Smith
   Great Lake Swimmers
   Iron & Wine
   Jack White
   Neil Halstead
   PJ Harvey
   Stevie Wonder
   The Doors
   The Kinks
   The Who
   Black Rebel Motorcycle Club
   Bruce Cockburn
   Cult With No Name
   Dave Matthews Band
   Death Cab For Cutie
   Dengue Fever
   Depeche Mode
   Elephant Revival
   Grateful Dead
   Harry Manx
   Jeff Beck
   John Lennon
   Joni Mitchell
   José González
   Loreena McKennitt
   Lucinda Williams
   Ludovico Einaudi
   Mark Knopfler
   Muddy Waters
   Neil Young
   Oi Va Voi
   Other Lives
   Paul Simon
   Ray LaMontagne
   Rilo Kiley
   Silversun Pickups
   Son Volt
   Stone Roses
   Suzanne Vega
   The Police
   The Pretenders
   Tom Waits
   Wye Oak
   Younger Brother
   Afro Celt Sound System
   Amy Winehouse
   Antje Duvekot
   Bat For Lashes
   Boy & Bear
   Broken Bells
   Broken Social Scene
   Caro Emerald
   Cult with No Name
   Danger Mouse & Daniele Luppi
   Doc Watson
   Duke Ellington
   Eric Clapton
   First Aid Kit
   Gillian Welch
   Gregory Alan Isakov
   Gustavo Santaolalla
   Jefferson Airplane
   Jenn Grant
   Jenny Lewis
   Jimi Hendrix
   Jimmy Cliff
   Joe Strummer & The Mescaleros
   Johnny Marr
   José González
   KT Tunstall
   Kate Miller-Heidke
   Kathleen Edwards
   Keren Ann
   Laura Marling
   Leo Kottke
   Lost In The Trees
   Massive Attack
   Medeski, Martin & Wood
   My Morning Jacket
   Old 97s
   Paul Weller
   Rebecca Zapen
   Robert Plant
   Sarah Blasko
   Sarah McLachlan
   Sophie Zelmani
   Tame Impala
   The Barr Brothers
   The Byrds
   The Cinematic Orchestra
   The Jam
   The National
   The Republic Tigers
   The Slow Show
   The Smashing Pumpkins
   The Wailin’ Jennys
   The Wallflowers
   Thea Gilmore
   Toad The Wet Sprocket
   Toots & The Maytals
   Van Morrison
   Vienna Teng
   1 Giant Leap
   A Fine Frenzy
   Alpha Rev
   Amadou & Mariam
   Amy Macdonald
   Ane Brun
   Anni B Sweet
   Aretha Franklin
   Barra and Govinda
   Basia Bulat
   Big Country
   Big Head Todd & The Monsters
   Billie Holiday
   Blood Sweat & Tears
   Bohren & Der Club Of Gore
   Buffalo Springfield
   Cat Power
   Choir of Young Believers
   Chris Whitley
   Christians and Lions
   Claude Chalhoub
   Cloud Cult
   Creedence Clearwater Revival
   Crosby Stills & Nash
   Dandy Livingstone
   David Byrne & St. Vincent
   David Gilmour
   Derek and the Dominoes
   Elliott Brood
   Elton John
   Emily Barker & the Red Clay Halo
   Fiona Apple
   Flaming Lips
   George Harrison
   Glen Hansard
   Greg Laswell
   Grizzly Bear
   Gus Black
   I Am Kloot
   Ian Moore
   Imogen Heap
   Ismael Lo
   J.J. Cale
   Jackson Browne
   James Taylor
   Jenny Lewis & The Watson Twins
   Jesse Sykes & The Sweet Hereafter
   Joan Osborne
   Joe Bonamassa
   John Cale & Brian Eno
   John Hiatt
   John Martyn
   John Mellencamp
   Kate Bush
   Little Feat

It could be interesting to do this kind of analysis with soma fm. Their stations–at least all the ones I’ve checked–post their playlists to Twitter, which gives us a built-in API for importing their playlist feeds. It might also be interesting (or it might not) to use’s API to pull a list of user-contributed tags for each song, and analyze radio stations that way (or use the data to recommend radio stations to users).



REVEALED: The reason for the Pope’s resignation.

Objective To explore the perceived wisdom that papal mortality is related to the success of the Welsh rugby union team.

Results There is no evidence of a link between papal deaths and any home nation grand slams (when one nation succeeds in beating all other competing teams in every match). There was, however, weak statistical evidence to support an association between Welsh performance and the number of papal deaths.

Obviously, fear of Welsh rugby ended Benedict XVI's papacy.


I finally got around to experimenting with Shiny, RStudio's new(-ish) web application framework.

My first app just generates a simple plot from the Social Security baby names data aggregated by Hadley Wickham.

I'm really impressed with how easy it is to develop a working app in Shiny.  If you already have a script that generates a plot or some analysis of your data, less than 50 lines of code can give you a simple, web-based interface to explore it.


Reading Images in R

I've been experimenting with the ReadImages package to analyse JPEGs in R, largely following this is-r() post

Most of the code behind this is in this gist, which you can also see below.

Rather than use k-means clustering, I've been experimenting with translating the colors in images into human-descriptive names. For example, what I have now can take this image:
Say Digital Humanities

And generate a plot showing the frequency of different colors in the image:

sayit <- codeColors("")
# A sample of what this data looks like now:
print(xtable(head(sayit)), type = "html", include.rownames = F)
Y X R G B rgb rounded code
-1 1 0.90 0.82 0.61 #E6D19C #FFCC99 white
-2 1 0.79 0.71 0.48 #C9B47B #CCCC66 yellow
-3 1 0.80 0.72 0.47 #CDB879 #CCCC66 yellow
-4 1 0.83 0.74 0.47 #D3BC78 #CCCC66 yellow
-5 1 0.82 0.73 0.43 #D0B96D #CCCC66 yellow
-6 1 0.85 0.75 0.44 #D8BF6F #CCCC66 yellow


Unnamed-chunk-1 (1)

It can also generate images showing the intermediate steps in generating the plot, such as this image reduced to 216 colors…

plotImage(sayit, 'rounded')


And this one, using the color names to color the pixels, instead of the much more specific color codes:

plotImage(sayit, 'code')


The color data itself is a csv list of the 216 “web-safe” colors with each color's “name” as I perceived it while looking at swatches on a web page, for example:

print(xtable(head(read.csv("")), n = 10),
type = "html", include.rownames = F)
colorID colorCode
000000 black
000033 blue
000066 blue
000099 blue
0000CC blue
0000FF blue

So far, the main point of failure is in images with a lot of white and/or shadows in them—shadows tend to have a purplish tinge, and a solid white wall can appear as a variety of colors depending on the light. Effectively, this problem results from the fact that our perception of color is heavily influenced by context–a color that looks purple or grey when you have a large rectangle of it on a web page might be clearly white in a picture of the Providence Athenaeum's interior, resulting in something like this image

Providence Athenaeum by Ken Zirkel

being perceived this way:

library <- codeColors("")
plotImage(library, "code")