Are recommenders reaching their limit?... No!

At the recent UMAP 2009 conference, a paper raised the possibility that we are reaching the possible performance limits of recommendation systems (RS). If true, this would change the landscape for research and development in RS. In fact, some blogs discussed this paper before it was even presented! However, after reading the paper, I'm a bit inclined to disagree at the hype over this paper. It's true, the paper does point to the performance limit for RS based on the current system of obtaining recommendation data. However, it does not mean that no one can build a better RS.

First, I want to discuss what the potential impact could mean for RS if indeed, we reach a true limit of performance. As an example, assume that for a particular task (e.g., music recommendation), people have a self-agreement 0f 90%. That is, a person will agree with themselves 90 times if they rank 100 songs one day and then rank the same 100 songs two weeks later. Assume that tastes do not change, which the authors argue is the case in their setting (they make three measurements at different points in time). What does this mean? Some possible explanations:

(1) The user doesn't know if or how much he likes the movie.
(2) The user doesn't understand or can't specify the degree to what he likes the movie into discrete, deterministic categories.

(1) is, by default, the wrong option since the user's judgment is the correct answer automatically; however, (2) makes some sense. While people may have an understanding if they really like or hate something, there is a rather large ambiguous area in the middle. How many people can consistently listen to a song and say, "I like that song 40%"? What does that even mean? Does the user like it 40% of the time he hears it? Does it mean that it would be in the 40th percentile of songs if the user were to rank every song he has listened to? If he ranked every song he's heard several times, would the average rank be the 40% percentile? The authors of the paper demonstrate this when they show that the inter-subject disagreement occurs 34% between rankings 2 and 3 and 25% of the time between 3 and 4 on a 5 point scale. In other words, people aren't able to rank movies accurately if they do not have a strong opinion. Usually, it assumed that the fault lies with the user; that is, a person is confused about what the categories imply. I disagree. I believe that it's a probabilistic rating because yes, opinions constantly change. People are not machines. They have emotions. Emotional states have an impact on how we both interpret and want to interpret our environment.

Second, how can we measure the success of a RS when 100% is theoretically impossible? What does this even mean? This issue has come up several times in terms of genre recognition. Until the reprint of Scanning the Dial and the accompanying criticism directed at the MIR community, some authors have validated their algorithms by stating that it is more accurate than humans. As pointed out in a couple papers, this is nonsensical since genres are ill-defined. Ultimately, our categorical dimensions of music is largely subjective and built over a life-time of (often conflicting) feedback from society. Still, we can ask, what if a RS comes out with a better accuracy than the documented limit? Does it know what people will like more than humans? Of course not. It shows an error in the choice of evaluation criteria. Ultimately, a RS is measured at a moment in time. If a person likes something on Tuesday, but does not like it on Wednesday, it does not mean the user is confused. It means he liked it on Tuesday, but not on Wednesday. Tastes may change based on mood, evaluation of new information based on the world around us, etc. Future RS may be able to detect this information to update adaptively.

This again brings up the problem with an RS. Every RS is based on the idea that a user will like something similar to what the user liked in the past. Further, almost all RS model a user as a single entity or that the user must maintain separate profiles for different tastes. For example, Last.fm and Pandora cannot build a station or user profile that maintains two separate personalities - it's up to the user to construct this system. Netflix only allows one user profile per account. While my fiancee and I may like some of the same movies, we certainly do not like all of them. Heck, some days I want a good skeptical show like Bullshit, but on another day I may want pure magical fantasy.

Even with transparency, somethings get muddled by small clusters of users who have a very demonstrated behavior. For example, Netflix is currently telling me that I will like Wallace & Gromit because I like This is Spinal Tap. How are these two even connected? The first is in the category "Children & Family Suggestions" and the other is a movie about a fictitious failed hair-band. Granted, both are good, but the only relation here is only based on discrimination (Wallace & Gromit is a UK show and Spinal Tap is an American movie about a British band). Apparently, all British humor is the same to Netflix users. The weirdest might be that I'll like a Talking Heads concert because I like the movie Fargo. Obviously, content analysis would do a better job of filtering the nonsense predictions.

So, in conclusion, research in recommendation systems is not reaching it's limit in performance. Rather, recommendations based on the idea that a user is a simple, static classifier is limited from the start. Smarter, better recommendation systems that can understand complex user behavior such as emotional state, thoughts on quality, and incorporating content analysis will be the building blocks of the next generation of recommendation systems.

Temporal Tag Information

Recently, I presented a paper at ICASSP that discussed the importance of incorporating temporal information into the structure of automatic tag recommendation algorithms. Until this paper, all studies and systems designed to overcome the "cold start" problem ignored temporal information for the most part. In fact, I am aware of only three exceptions:

(1) Using derivatives of features, such as MFCCs. This incorporates temporal information on only a small scale (~50-100ms).
(2) Averaging features from multiple frames in a given temporal window (ex: averaging 100 ms frames over a duration of 1 second).
(3) Extracting song-level features, such as rhythmic features; e.g., estimated beat histograms.

The problem with the above approaches it that they incorporate information on a very small scale and do not incorporate "syntactic structure." One problem I have had with music recommendation research is that many of the systems are based on a rather faulty assumption. Researchers have taken an abstract from a presentation given by Gjerdingen and Perrot in 1999 and essentially taken the results way too far. Specifically, researchers have taken for granted the "bag-of-frames" approach, which essentially says that any small segment of a song is representative of the whole song. In other words, one can listen to 250 ms of a song and that will be representative. This is obviously a faulty assumption and it has been discussed here and here. Originally, this assumption was used in studies on genre classification. Since genre is an ill-defined concept anyway, it is difficult to verify this assumption. However, even if this assumption is true, it does not make sense to translate this to tags, which have a better defined meaning. For example, the Pandora tags of "repetative melodic phrasing" and "extensive vamping" have obvious acoustic semantic structure.

So how does our study contrast with previous approaches?

In the paper, we build a vocabulary of acoustic tokens, which can be seen as acoustic generalizations of phonemes in automatic speech recognition (or musiphones as Doug Turnbull called them - yes, Doug, I consider this your terminology). Not only are the musiphones represented by a temporal model (i.e., a multi-state HMM), but syntax is also considered through the use of unigram and bigram counts. Compared to the baseline (Turnbull, Barrington, Torres, and Lanckriel), our algorithm performance substantially better; especially for tags which are considered to be temporal in nature (i.e., melody, solos, etc.). While this paper mirrors the implementation we proposed for genre detection in 2006, the results are more informative in the 2009 ICASSP paper.

Note: The Gjerdingen and Perrot presentation has finally been published so that people can read the study in its entirety. Two things to note in addition to the papers linked above:

(1) Gjerdingen and Perrot performed a task of discriminating genres (i.e., a closed, forced choice) and not identification (i.e., an open, forced or unforced choice).
(2) Only 10 genres were used and most where fairly easy to discriminate.

ICASSP Day 2

Today was a much more interesting day in terms of technical content (at least, to me). In the morning I went to Content-based Audio Processing session (at least the first half). All the presentations were good, but I liked "Interpolating Hidden Markov Model and its Application to Automatic Instrument Recognition" by Tumoas Virtanen and Toni Heittola the best. Essentially, the motivation is that hidden Markov models (HMMs) make an assumption that an audio element is decomposed into discrete states. In reality, there would be a smoother transition between states and while one can add states to the model to reduce the error, it requires a lot more data since there are more parameters. Their suggestion is to create auxillary states by using interpolation methods on the HMM parameters. They demonstrated a 5% absolute improvement over the baseline (no interpolation). However, their test database was isolated instrument recognition and I would have liked to see how their approach behaved in the presence of different noises and levels. Using discrete states can help with noise reduction by vector quantizing the state space. It's possibe that the presence of noise could take the state trajectory down an erroneous path. For the afternoon, I went to the poster sessions and there's plenty I could say, but I'd miss the rest of the conference. Instead I took pictures of the posters and they are presented below.

This is a plug for Emiru Tsunoo's paper titled "Rhythm Map: Extraction of Unit Rhythmic Patterns and Alanysis of Rhythmic Structure from Music Acoustic Signals." I've read the paper and it's very good work. I like the search for new and better features.







Emiru presenting his poster. He was crowded the entire time.








Dr. Sagayama explaining RhythmMap to Malcolm Slaney of Yahoo! and Stanford.

ICASSP!

From Japan to Taipei! I have a back catalog of pictures to post from my visit in Japan, but I figured that I should cover ICASSP in "real time." Yesterday was the opening ceremony, which was actually quite nice. I forgot my camera, so I'll have to track down pictures of the opening musical act, which contained live authentic Tawainese music. I'll try to post everyday, but no promises. The ISMIR deadline is soon afterwards and I may submit a couple of papers, which means that ICASSP is a working conference. However, I will take a minute to say that I liked the proceedings.

My new diggs (for about a month)

Finally settled here in Tokyo. Here is my new home away from home. Definitely sparse, but hey, I really just need a laptop with Internet. The people here have been incredibly nice and have been very patient with my terrible impression of someone trying to speak Japanese. Yes, I'm that bad that it would even be bad if I was trying to speak it poorly.

It was definetly one of the worst plane rides I've ever had. As soon as we got in the air, we were told that we had to land in Denver because of a medical emergancy. We didn't get back in the air until many hours later. Special tip for anyone: if you feel like you are having problems breathing on the ground, it's not going to be better 5000+ feet in the air. Special thanks to Uchiyama Yuki for staying awake to get me to the hotel.

I have tons of work to do here, but it should still be a fun and very rewarding trip. More tomorrow.

Exciting Oppotunity

Ohayoo gozaimasu, konnichiwa, or konbanwa to translate James Randi's standard greeting on his "Randy Speaks" segments. I've been putting off making this announcement because, well, I've been very busy and exciting "of the moment" things needed comment first. I will be traveling to Japan to be a visiting student for a few weeks at Dr. Sagayama's lab at The University of Tokyo. It is a great privilage and honor to achieve this opportunity. One project I am excited about is automatic chord detection, which I have been working on for my thesis. Dr. Sagayama's group did very well in last year's MIREX competition, but my adviser and I have already noted a couple areas for improvement. I am also excited to learn about the many cool things Dr. Sagayama's lab does.

During my long stay there, I will be giving a talk on "Acoustic Segment Modeling for Music Information Retrieval" and how semi-supervised and unsupervised training can bridge the missing gap between automatic speech recognition technology and music information technology. I will briefly discribe the importance of separating the contextual and interpretive nature of music from acoustically grounded attributes when training acoustic-based systems. I am finishing up the slides now and will post them after my talk.

Anyway, during the next three weeks, you may see some pictures of Japan and Taiwain since I will be going to ICASSP to give another presentation titled "On the Importance of Modeling Temporal Information in Music Tag Annotation." Sayoonara!

Malcolm Slaney Talk On Friday

Malcolm Slaney will be giving a talk in the TSRB building at Georgia Tech on Friday. Unfortunately, I may be out of town since my soon-to-be sister-in-law is getting married the week, but I'm trying to talk my fiancee into going. So if you are in the Atlanta area, check out the talk and give me some notes!

Details:

"We're drowning in Multimedia. Hurray!!!!"
Malcolm Slaney
Yahoo! Research and Stanford CCRMA

Friday, March 13th, 11 am
TSRB 132


The wealth of data available on the Internet changes the way we think about multimedia. Never before has there been so much multimedia data available for training models and answering questions. But these new riches bring with it a change in the problems we must think about. The data is noisy and largely unlabeled --- we must make sense of it, often returning an answer in hundreds of milliseconds. How do we understand the user's environment, especially when it extends across the world? How do we take into account context and do it at the scale of the
Internet? In this talk I'd like to share with you Yahoo's experiences in this brave new world of multimedia everywhere, describe promising new technologies, and discuss open research directions. I will describe the need for better user and multimedia models, the kinds of algorithms needed for today's large databases, and how the Internet is changing multimedia retrieval.


Biography

Malcolm Slaney is a principal scientist at Yahoo! Research Laboratory. He received his PhD from Purdue University for his work on computed imaging. He is a coauthor, with A. C. Kak, of the IEEE book "Principles of Computerized Tomographic Imaging." This book was recently republished by SIAM in their "Classics in Applied Mathematics" Series. He is coeditor, with Steven Greenberg, of the book "Computational Models of Auditory Function."

Before Yahoo!, Dr. Slaney has worked at Bell Laboratory, Schlumberger Palo Alto Research, Apple Computer, Interval Research and IBM's Almaden Research Center. He is also a (consulting) Professor at Stanford's CCRMA where he organizes and teaches the Hearing Seminar. His research interests include auditory modeling and perception, multimedia analysis and synthesis, compressed-domain processing, music similarity and audio search, and machine learning. For the last several years he has lead the auditory group at the Telluride Neuromorphic Worksho.

MIR Student Researcher in Press!

Congrats, Anita on the excellent CrunchGear writeup!  It appears that MIR research is starting to pick up steam in terms of popular opinion.  Don't forget to read the comments of the article.  Now, get back to work!  The people want more!

Old Copyright Laws Hurt Research

Note: Thanks to my brother, Josh, for his comments.  Josh is an IP lawyer in Chicago, IL.

Recently, a question was phrased on a research mailing list, that more or less went as follows: the researcher was conducting a listening experiment and there was a potential that the subjects could potentially find and keep the 15-second excerpts for personal use.  The author was worried that this constituted a copyright violation.  I pointed out that more than likely this falls under fair use.  However, reading this gives one clear impression: the law itself is rather meaningless.  First, the law only stipulates what needs to be considered in evaluating fair use, without giving guidelines or specifics.  The webpage states "There is no specific number of words, lines, or notes that may safely be taken without permission," and that it is best to obtain permission from the copyright holder.  Further, the precedent given only gives a partial list of examples that was relevant in 1961.

These points are key to researchers in information retrieval (and in particular, music information retrieval) because these laws were based on the 1960's technology.  Simply put, exchanging songs, text, images, etc., was a rather involved task.  Today, the exchange and storage can be conducted on a massive scale, unforeseen by the lawmakers fifty years ago.  With this increased capacity for storage, researchers can now test large-scale IR algorithms and the need for a (relatively) free, large scale database is needed.  However, in the case of music, such large scale databases are impossible to find or have severe restrictions on them.  Every year, I see experiment after experiment of promising algorithms, but results must be taken only so far because of the size and scope of the testing database.  Even though some schools have access to a large library archive of recordings, researchers at other institutions are unable to duplicate their results because the data is not freely available.

Some researchers have found "loopholes" that allow them to share features extracted from audio, which cannot be used to recreate the audio (e.g., Mel-cepstral coefficients).  This is still not a viable solution because no-one can a priori determine the best features for all IR experiments and experimentation with new features is impossible.  Also, potentially, a set of features, which in combination may be reversible could lead to the best results, but this is impossible to test if only a limited set of features is ever distributed.

A very interesting solution comes in the form of MIREX, where a TREC-like evaluation is conducted by having researchers send in algorithms to various competitions.  However, there are a few drawbacks.  First, it is an enormous burden on the sponsoring institution, IMERSIL at The University of Illinois.  The livelihood is also completely depended on the program's funding, which is fine for the next few years, but the long-term stability is not guaranteed.  Second, the evaluation is carried out once a year, but there was talk of extending this to a rolling model.  A third problem is that tasks are largely fixed and a new task is only considered if it has broad approval.  New and interesting tasks are still subject to small, private databases before their inclusion in the task.

I applaud those at IMERSIL for coming up with the proposed solution and also those that supply databases in some form or another, but these are patches to the main problem, which, as I have stated, is that copyright regulations are severely out-of-date.  Simply put, when today's regulations were implemented, no one imagined the scalablity of today's information age.  Regulations are not only needed for the public sector to address today's file-sharing "problem," but also, better regulations are needed for today's researchers.

The problem ultimately stems from the current practice of common law.  Simply put, our current laws are written as loose guidelines and the specifics are left open to the courts.  Despite what you learned in history class, our laws are not actually written by legislatures, but rather by those on the bench.  Look at The Sherman Act: a single sentence determines when the law is applicable; however, courts have expanded and contracted this law as they see fit.  Instead of a coherent, well-structured law that anyone can follow, one needs a swarm of lawyers to get through any issue.  Worse, many people are completely unaware that they may be breaking copyright law.  Many researchers wrongly assume that if they use less than 30 seconds, then they are legally safe, but this is untrue.  It is purely dependent on whether the recording industry chooses to go after you and how good your defense team is.

So what would a good solution look like?  I have thought of one that is actually rather easy and is found in other research fields.  Handling of nuclear, biological, and chemical materials contains a strict set of guidelines for researchers to follow in obtaining, handling, and destroying potentially dangerous chemicals.  I'm actually a little surprised that a similar structure has not been suggested for the use of copyright materials.  Such guidelines could allow researchers access to large amounts of complete, unaltered data (i.e., full songs, raw audio), while still ensuring the rights of the copyright holders.

I can already address the objection that will be raised by the copyright holders: "But very few researchers will want to take home nuclear, biological, and chemical materials."  This is just untrue.  Many research labs conduct studies on illegal drugs, such as marijuana.  Are you telling me that no researcher would want to take home a little stash?  Again, strict guidelines are in place to ensure that researchers use these illegal substances in an ethical and legal manner while also ensuring that necessary research can be conducted (http://www.safeaccessnow.org/article.php?id=5183).  This is definitely possible in terms of music, text, and other multi-media.

FASTLIB/MLPACK

I talked to a couple people at ISMIR about a new machine learning toolbox, called FASTLIB (although, it appears to be called both FASTLIB or MLPACK). This toolbox was developed by Alexander Gray's lab in the College of Computing at Georgia Tech and I used this extensively in Alexander Gray's class. I highly recommend that anyone try this toolbox for their machine learning needs. Programming within the guidelines greatly reduces the programming time (almost to the simplicity of MATLAB), while retaining computational speed and memory capacity. If you are like me, how have had to make the judgement call between programming something in MATLAB and having it run a long time, or spending a long time writing and debugging C++ code so that the algorithm runs quicker.

The official place to download the package seems to be here; however, I found some issues (expected with a version 1.0). The stripped down package on an old class website seemed easier to install. The individual built-in algorithms can be added manually later. I hope to have a small series of posts demonstrating the ease of programming style.

MIR and The Media: How do we interface?

The music information retrieval (MIR) community has seen an increasing amount of press time in the past couple weeks (see here and here).  As this type of research gets more press coverage, an important issue is how the researchers interact with the press.  Recently, my favorite podcast, The Skeptic's Guide to the Universe interviewed Sharon Begley, the Senior Science Editor at Newsweek, about how science and media interact (podcast #166).

First, she discussing that relying on journalists to get the word out is sure-fire way to never get any press.  Surprisingly, for someone in "Big Media," she applauds The New Media paradigm.  Self-promotion, however, raises many ethical considerations since a few researchers are apt to over-publicize their results and well before any peer-review process has taken place.  I think it is key that researchers limit their publicity until after some form of peer-review.  For example, you will not hear any results about my research on this blog until it has been approved for publication. 

Second, the press is a profit-making machine.  Therefore, the fantastical will always get more press than the consensus.  The same pressures that apply to other media types, especially in the face of The New Media.  Generating readers will be the main point of any major media outlet.  Generally, the story will be modified to be mostly true, but the key point may still be lost.  It is up to the researchers to keep their message intact.

Third, researchers cannot expect the press to understand everything in their.  At best, you can hope for some amount of scientific knowledge from the press, but they will always trust a Ph.D. in some science, even if the guy is crank (e.g., creationists, homeopathy, etc.).  Being able to describe research in comfortable layman's terms is an essential skill for any scientist/engineer.

Preparing for ISMIR

I am getting excited about ISMIR in a couple weeks. As many people suggested to me last year, I am going to ISMIR with (almost) nothing to do but take it all in. I do have a late breaking session poster, but that is on the very last session on Thursday. I do not anticipate many people attending the session since most will be trying to make flights back home. To be honest, I think they should scrap the last day or make it a full day with the dinner to end everything.

I have even started reading some of the proceedings as Elias pointed out that they are up for everyone to see. So far I have read about five papers. I have to admit, I was a little fearful about the focus of interdisciplinary research being incorporated into every individual paper. This is largely subjective and I feared that people would apply too broad of a focus and the papers would not get into technical detail in any one area. While I am sure some of this is true, I was happy to find a couple papers that were really good. I liked the paper by Moh and Buhmann about adaptive kernels and would like to see it applied to something other than artist classification, which does not necessarily translate into general similarity as Elias pointed out in his thesis. Matthew Riley, et. al., was good too. It is great to see that people are tokenizing songs to incorporate dynamics better. I think they could get more modeling power if they added HMMs and did something closer to the acoustic segment modeling approach that my adviser and I did at ISMIR a couple years ago. I really like Kurt Jacobson's paper on identifying artist communities in social communities, especially the attempt to incorporate audio analysis into the design. I am definitely going to discuss this with him at the conference.

If I have not mentioned your paper, then I probably have not read it yet, so do not take offense. I will get to it and I am sure I will like it, even if I do not post about this in the future.

Closed-set vs Open-set Tags

Ugh. The cluster is still down. I was hoping to get something together for MIREX's tag annotation contest, but there is no way I can get to it with everything else I have going on. Oh well, maybe next year. Anyway, on to the subject.

I have been examining playlist prediction using Last.fm and Pandora tags. Not surprisingly, I got this result:


This was a real simple nearest-neighbor search. While this gives evidence to (part of) my hypothesis that Last.fm's "anything goes" open tag set will perform better than Pandora's expert-assigned closed tag set, I need to eliminate some other variables before any final conclusions are made. Most noteably, Pandora's tag set has a size of around 500 tags, while Last.fm's tag set is (at least) on the order 10,000. In fact, on just a subset of the USPop set, I found over 20,000 tags. I need to reduce the dimensions so they are comparable, but still maintain the flavor of Last.fm's set.

Occum's Razor and the "Rap Problem"

Yesterday, I briefly described the "Rap problem," which is where artists names appear several times in a database because they feature other artists. It's probably unfair to "pick" on rap after looking at the greatest violators, but there is a clear trend that rap is a fairly big violator. Note: I'm not saying rap sucks or anything like that. I'm just saying that this presents a problem for researchers dealing in search technology. In fact, as I'll show, people who feature lots of guest artists make a pretty impressive list of musicians and performers.

At first, I thought I would have to do an extensive literary search for an efficient solution to this problem, but my girlfriend proposed a quick solution. She suggest that I just look to see if the artists' names are the first ones listed. At the surface this seemed reasonable, except that some artists have names that are sub-sequences of other artists (e.g., "Joe", "Pink"). But this lead to an efficient solution to the problem: look for names that are equal or that have a special formatting. For example, most of the feature problems can be dealt with by looking for the regular expression /^artistsName_feat_/ or /^artistName_&_/ (underscores and ampersand are not wildcards).

This actually worked fairly well since I am only looking for a group of users that listened to songs from my dataset. This is not a solution to the misspelling problem, but it's a fair assumption that most people will listen to correct spellings when using a well-established site like Last.fm. This greatly saved some time and proved once again that one should always try something quick and dirty first.

Looking at the top 20, there is a definite pattern:

mariah_carey:	135
busta_rhymes:	105
usher:	54
nelly:	52
madonna:	48
ludacris:	42
wyclef_jean:	40
santana:	39
michael_jackson:	37
bob_marley:	37
david_bowie:	35
ja_rule:	32
dmx:	31
nelly_furtado:	31
ricky_martin:	29
frank_sinatra:	29
sting:	28
cypress_hill:	27
elton_john:	27
outkast:	25

One should note that artists like Mariah Carey and Busta Rhymes have not necessarily played with over a hundred different artists because those artists can have different spellings, which I did not correct for (e.g., "mariah_carey_feat_boys_2_men" vs "mariah_carey_feat_boys_ii_men). However, the likelihood of mispelling of the featured artists is probably not an inherint trait of the first artists, so we can treat it as noise. I don't think Mariah Carey has a particular fondness of easily mispelled or varied names.

One can also divide this group into about 3 groups (some overlap depending on personal genre definitions): hip-hop, rap, and old and established rock/pop artists. So, the "rap" problem may not be such a problem in terms of taste given the list above. Also, voice and style are very central to the "musicalness" of rap and hip-hop, so using a different artist is probably the same as a rock musician using an orchestra or a different instrument than normal.

The Continued Popularity of USPop2002

In order to gather some useful training data for my thesis, I need to get some preference rankings for music recommendation. It is also necessary for there to exist tag information as well, such as Last.fm and Pandora. Further, I must be able to obtain audio (or some acoustic features) rather cheaply. The best data I have found is LabROSA's USPop2002. It's much larger than RWC Database and because the songs are based on popularity in 2002, it is much more likely to have tags than Magnatunes. The downside is that I'm limited to Mel-frequency cepstral coefficients.

While, LabROSA also has playlists from OpenNap, there are no preferences given; a song is either on a person's playlist or not on a person's playlist. I've been using Last.fm's API to try to remedy this situation. First, I gathered the top listeners for each of the 400 artists in the USPop2002 set. Over the past couple weeks I have been extracting the total combined weekly chart lists to get the number of plays of a particular song for each listener. While number of plays may not be a direct measure of preference (or rating), it is reasonable to assume that people will listen to song they like more than the ones they do not like. At the moment, I have only downloaded about 4000 listeners (I have to download several pages per listener and Last.fm requests a 1 second wait between requests). Also, artist names appear in several different varieties. Rap and hip-hop seem to be exceptionaly bad since they are unable to do any song without a guest star.

There's tons of data to play with, but for now, let's look into what artists are popular. Note: there are still thousands of users to download and some artists' top 50 listeners have not been reached yet. These results should be taken with precaution so that we don't leap to Montauk monster conclusions (it's a racoon, let it go people).

This kind of continued success is what I would expect to see: superstars make up the vast majority of hits and the short-lived fame of others dies out. However, one should note the artists appearing at the bottom may have more plays due to the "rap problem" described above. I also wanted to see if the data was consistent with Zipf's Law, but it is not (the bend is not deep enough).


One neat thing occurred in the top 5 artists: Beatles, Radiohead, Pink Floyd, David Bowie, Queen. Only the Beatles and possibly David Bowie have had enough users from their lists to explain such high results. Indeed, it appears that the other artists would be just as popular if I had taken a random group of users (note: I'm sure the Beatles will also have this once I extract more pages).

I'll have more later.

X-cluster down

The X-cluster was taken down today for summer maintenance. Looks like it may be two weeks, but hopefully the file server gets back on-line soon. I'll probably post some preliminary results on a couple of experiments during this time.

NPR API

National Public Radio has released its API. It already looks to be a tremendous research because they have audio content. I'm new to APIs so I'm pretty jazzed about this. I've only gotten to play with it for about 5 minutes, but I have verified that the audio is great. One potential application I foresee is music/speech detection and segmentation. Also, on the speech side, this data is great for topic identification. I'll hopefully have more to say on this later, but for now, I've got to go. It's 8PM on a Friday and my girlfriend is telling me I have to stop working.

MIR Group on CiteULike

I have started a group on CiteULike for music information retrieval researchers focusing on similarity and retrieval from audio. This is to allow us to see what papers others are reading on the subject. The focus is on using non-symbolic audio as the original format. For example, using MFCCs to build genre-level Gaussian mixture models is relevant. Using DTW on MIDI signals is not relevant unless the MIDI signal is a mid-level representation (ex. "Specmurt analysis"). Onset detection is not relevant; however, using onset features to classify dance music is relevant.

I greatly encourage other fields to start their own groups (I may also start more if others join). I felt restrictions on the scope of the group was important because MIR is becoming too broad of a field. I expect that many researchers may be in several groups, which is great and there may be a lot of overlap in the papers appearing in these groups. However, in our "Everything is Miscellaneous" world, this is not a bad thing.

I've restricted that new users must be approved, but this is simply to generate a list of who's who. Anyone that wants to get in will be accepted. I am also willing to free up restrictions on anyomous postings if people want, but I want to prevent abuse since this supposed to be useful and non-combative.

New Project

I'm working on a new project in language identification. Specifically, we are looking into using speech attribute detectors to enhance phonetic transcriptions. From there, supervectors are created by creating phone document vectors for each language. Moreover, we are using TempoRAI Patterns (TRAPs) as features. These have been shown to be superior to using MFCC + velocity + acceleration vectors. I would be interested to see how these perform on music, especially since incorporating dynamic features have had only limited effect. I think part of the problem is that music is (generally) slower than speech, so incorporating longer windows might be better. TRAPs are also different from texture windows because texture windows are simply first and second order statistics from the frames within the window, whereas the original frames are concatenated in TRAPs. However, since I'm limited to using USPop's feature set (MFCC), I'm not sure I'll get to see the effect any time soon.

Name this tune...

An interesting paper appeared in Psychology of Music, titled "Memory and metamemory for songs: the relative effectiveness of titles, lyrics, and melodies as cues for each other" by Pyrnircioglu, Rabinaovitz, and Thompson. There findings indicate that while people cannot remember lyrics well when given a title or melody, lyrics are better to remember titles or melodies than using titles or melodies to remember the other. However, if someone couldn't remember the target with certainty, they were asked to pick one of four choices and then asked how sure they were. In this case, lyrics were seen as not much help when used to try to remember a melody or title, even though they scored best when used. Also, even though lyrics were the never really remembered given a melody or title, people picked their choices with more certainty.

I find this pretty fascinating, but it would be interesting to see an additional study: the roles of tags and non-acoustic information. Many content-based retrieval algorithms are bootstrapping their acoustic classifiers with textual descriptions (e.g., tags). The basic idea stems from websites such as last.fm. However, I've never seen that these tags remain universal in meaning. For example, given that a song is listed with the tag "grunge," can we safely assume that everyone would understand this? Or are tags only valuable to the person that assigned them? It's probably somewhere in the middle, like genres. However, give enough tags, we can get a good "picture" of what the song contains.