Joshua Feldman

Elbow plot

To help us decide how many clusters to choose, we can use the elbow plot technique. Unfortunately, the elbow plot exhibits a smooth curve that does not make it easy to decide how many clusters would be appropriate. Nevertheless, 10 seems an appropriate number, since it sits somewhere in the middle of the elbow region of the plot.

# Set seed for reproducibility
set.seed(1234)

# Scale the data
beatles_core_metrics[2:10] <- scale(beatles_core_metrics[2:10])

# Use map_dbl to run many models with varying value of k (centers)
tot_withinss <- map_dbl(1:30,  function(k){
  model <- kmeans(x = beatles_core_metrics[2:10], centers = k)
  model$tot.withinss
})

# Generate a data frame containing both k and tot_withinss
elbow_df <- data.frame(
  k = 1:30,
  tot_withinss = tot_withinss
)

# Plot the elbow plot
ggplot(elbow_df, aes(x = k, y = tot_withinss)) +
  geom_line() +
  scale_x_continuous(breaks = 1:30) +
  geom_vline(xintercept = 10, linetype = "dashed")

K means

Now that we have decided on how many clusters to use, we can fit our k-means model by using the kmeans function in the built-in stats package.

# Set seed for reproducibility
set.seed(1234)

# Run the model
beatles_k <- kmeans(beatles_core_metrics[2:10], centers = 10, nstart = 100)

We can now assign clusters to each of the tracks and visualise how they differ in key attributes. However, since it is impossible to visualise all of the nine dimensions that we clustered them on, we can only compare two attributes at a time – for example energy and valence.

# Save the clusters
beatles_core_metrics$cluster <- beatles_k$cluster

# Plot energy against valence
beatles_core_metrics %>%
  ggplot(aes(energy, valence, color = as.factor(cluster))) +
  geom_point()

We can get a more detailed impression of the clusters by looking at the matrix of cluster centres. The sign and magnitude of these values give us all the information we need to describe our clusters qualitatively.

# Print cluster centres
round(beatles_k$centers, 2)

##    danceability energy loudness speechiness acousticness instrumentalness
## 1          0.07   0.50     0.20        0.55        -0.29             0.00
## 2          1.00  -0.67    -0.33       -0.31         1.10            -0.34
## 3          0.28   0.40     0.31       -0.46        -0.90             3.20
## 4         -0.81  -1.68    -2.39       -0.30         1.22             3.33
## 5         -0.34   1.10     0.66        0.31        -0.20            -0.30
## 6         -0.89  -0.11     0.09       -0.12        -0.75             0.02
## 7         -0.22   0.30     0.45       -0.33         0.05            -0.34
## 8         -0.38   0.20    -0.45        4.75         0.12             0.35
## 9          0.71   0.28     0.39       -0.31        -0.78            -0.32
## 10        -0.80  -1.69    -1.39       -0.29         1.44            -0.30
##    liveness valence tempo
## 1      2.84   -0.19 -0.58
## 2     -0.44    0.29  0.04
## 3     -0.09    1.03 -0.05
## 4      0.09   -1.81 -0.88
## 5      0.12    0.82  0.92
## 6     -0.35   -1.04  0.84
## 7      0.23    0.38 -1.06
## 8      0.30   -0.43  0.35
## 9     -0.46    0.54 -0.12
## 10    -0.59   -1.35 -0.22

Cluster #1: Performances

The Beatles performing ‘Get Back’ on the rooftop of Apple Corps at 3 Savile Row (Source: Jeffrey Pepper Rodgers)

• Key features: High liveness
• Number of songs: 12 (7% of total)
• Examples: Get Back; Sgt. Pepper’s Lonely Hearts Club Band; One After 909

With high liveness, Cluster #1 tracks give the impression that they were recorded in front of a live audience. They are energetic and written with the stage in mind.

Cluster #2: Ballads

Songs in Cluster #2 – which I’ve loosely called ‘Ballads’ – come under the broad umbrella of folk rock, often telling wistful tales about colourful characters or romantic interests. The cluster also captures some of The Beatles’ experiments with baroque pop (‘Eleanor Rigby’, ‘For No One’) and music hall (‘When I’m Sixty Four’, ‘Your Mother Should Know’).

• Key features: High acousticness, high danceability
• Number of songs: 27 (15% of total)
• Examples: In My Life; Eleanor Rigby; Michelle

Cluster #3: Interludes

The three tracks in Cluster #3 are statistical anomalies, perhaps too short for their musical features to be reliably expressed. Nothing much to see here.

• Key features: High instrumentalness, high valence, low acousticness
• Number of songs: 3 (2% of total)
• Examples: Birthday; Hey Bulldog; Words Of Love

Cluster #4: Instrumentals

Cluster #4 comprises the instrumental portion of The Beatles’ portfolio. Most of the Yellow Submarine album – the soundtrack to the animated film of the same name – is sorted into this cluster.

• Key features: High instrumentalness, low loudness, low valence
• Number of songs: 9 (5% of total)
• Examples: Pepperland; Sea Of Holes; Sea Of Monsters

Cluster #5: Rockers

Cluster #5 is classic rock and roll, with high energy, high tempo and an upbeat mood. It picks up many of the early Beatles classics (‘Twist And Shout’, ‘I Saw Her Standing There’, ‘Love Me Do’), as well as the band’s later experiments with hard rock (‘Back in the U.S.S.R.’). This is a coherent cluster that works well as a standalone playlist.

• Key features: High energy, high tempo
• Number of songs: 27 (15% of total)
• Examples: Twist And Shout; I Saw Her Standing There; Love Me Do

Cluster #6: Meditations

Several of the tracks in this cluster were penned by George Harrison and show an Eastern or psychedelic influence (‘Tomorrow Never Knows’, ‘Blue Jay Way’). They are often sad and meditative, whether contemplating love and devotion (‘Something’) or the loneliness of a spiritual hermit (‘Dear Prudence’). There is a skew towards The Beatles’ later work, with no tracks from the first four albums.

• Key features: Low valence
• Number of songs: 25 (14% of total)
• Examples: Here Comes The Sun; Something; Dear Prudence

Cluster #7: Slowcoaches

Cluster #7 is distinguished by its slow tempo, with languorous songs including ‘Dig A Pony’ (59 bpm), ‘Little Child’ (76 bpm) and ‘All My Loving’ (78 bpm). A fifth of the tracks are from the band’s second album, ‘With the Beatles’.

• Key features: Low tempo
• Number of songs: 25 (14% of total)
• Examples: All You Need Is Love; Yellow Submarine; Hello, Goodbye

Cluster #8: Vignettes

This small cluster picks up some of The Beatles’ more lyrically dense tracks, primarily characterised by their descriptiveness and emphasis on the spoken word.

• Key features: High speechiness
• Number of songs: 25 (3% of total)
• Examples: Strawberry Fields Forever; Polythene Pam; Maggie Mae

Cluster #9: Middlers

Cluster #9 sits somewhere between the hard rock sound of Cluster #5 and the folk rock sound of Cluster #2. It has no distinguishing features, with fairly average loadings for all nine of the acoustic features. It is the most populous cluster, representing 21% of The Beatles’ discography.

• Key features: None
• Number of songs: 38 (21% of total)
• Examples: Help!; A Hard Day’s Night; Eight Days A Week

Cluster #10: Lullabies

Cluster #10 is a collection of sleepy gems from The Beatles’ catalogue and works well as a standalone playlist (aside from the intrusion of ‘Her Majesty’ towards the end). Primarily characterised by a low energy, it is a nice assortment of songs to drift away to.

• Key features: Low energy, high acousticness, low loudness
• Number of songs: 12 (7% of total)
• Examples: Yesterday; Blackbird; Golden Slumbers