In this course you will learn about audio signal processing methodologies that are specific for music and of use in real applications. We focus on the spectral processing techniques of relevance for the description and transformation of sounds, developing the basic theoretical and practical knowledge with which to analyze, synthesize, transform and describe audio signals in the context of music applications. The course is based on open software and content. The demonstrations and programming exercises are done using Python under Ubuntu, and the references and materials for the course come from open online repositories. We are also distributing with open licenses the software and materials developed for the course.
Freesound API
Loading...
來自 Universitat Pompeu Fabra of Barcelona 的課程
Audio Signal Processing for Music Applications
168 個評分
從本節課中
Sound and music description
Extraction of audio features using spectral analysis methods; describing sounds, sound collections, music recordings and music collections. Clustering and classification of sounds. Demonstration of various plugins from SonicVisualiser to describe sound and music signals and demonstration of some advance features of freesound.org. Presentation of Essentia, a C++ library for sound and music description, explaining how to use it from Python. Programming with the Freesound API in Python to download sound collections and to study them.
與講師見面
Xavier SerraFull Professor
Dept. of Information and Communication Technologies, UPF
Prof Julius O Smith, IIIProfessor of Music and (by courtesy) Electrical Engineering
CCRMA
Hello. Welcome back to the course analysing or
processing for music applications.
This week, we are talking about sound and music description, and
in particular, we are talking about how to analyse sounds,
in order to then be able to describe them within sound collections.
So that we can compare sounds, we can classify,
we can cluster sounds and describe in that way collections of sounds.
Therefore, we need access to collections of sounds
from a software type of perspective.
So we need to be able to access the sounds programmatically, and
then obtain this analysis results and play around with them.
The Freesound API, that's an excellent tool for doing that.
So Freesound has a lot of sounds, and through the API, we can,
using Python for example, we can access all the information within Freesound,
and then process it in many different ways.
So let's try that.
So let's go first to the Freesound website.
Okay, and in order to learn how to use the API,
we can go to the developers section down here.
Okay, and here it tells me a little bit about the API.
So first of all, what's an API?
Well, an API is basically, it's a protocol, and
in the case of web APIs, it's a protocol that uses the HTTP protocol.
And we can make calls, HTTP calls to Freesound in a way to
query the specific content of what is in the database.
So we don't have to use the web type of interface to access that.
And in the documentation, there is information about how to actually do that.
So there is information about how to
access the different sources of information.
So if we go to that Resources page,
here it tells me how to search the different content types.
So for example here in search resources, it start by talking about a text search.
So we can query, and
we can do the standard text query that Freesound has, but programmatically.
And we can apply filters and we have basically access to all the information
that has a text label inside Freesound.
But we can do a lot more than that and we can also search for the content.
So we can search for example do the analysis data this low level features,
and then not only just get them, but filter them.
So for example here,
there is an example of how to get the mean
of the pitch of a sound that is 220 hertz.
Okay, so we'll find sounds whose mean pitch is 220 hertz.
So that's pretty good, can be quite powerful to do that.
To get a visualization of that, we can go here.
This is a specific call of the API.
Strictly speaking, it's not a call to the API because we are not
using the kind of a software to make this call, but
it's an option to have a URL
that displays the information of one of the resources of the API.
So if we specify the Freesound website, and
then follow by API Version 2, so API V2,
which is the API version we are using, and
then specify the sound and the number of the sound.
Every sound in Freesound has an ID, a number.
And it returns the information of this resource,
which basically has a summary of all the information of that particular sound.
So in here, we see some specific examples
of API calls that return more specific information.
But then, the default information is the one here.
So this is the actual call of the API that we are now doing.
And in here is the information that is returned.
So we have, for example, the core information of the sound,
its ID, its URL, the name of the file,
the tags that the user specified, the description of the sound, etc.
Whole bunch of things, when it was created, the license the bit rate,
the bit depth duration and also things like preview.
So we have access through the API.
So some low quality or high quality previews, or the images,
the display images that are shown in the Freesound.
So that's very good information that of course we can get it all together, or
we can just to the API ask for some specific items of these information.
And then are themes like also the analysis information or
the sounds that are similar to this one.
So we can also access that through the APIs.
So for example, let's click add the analysis statistics, okay.
So this returns the low level
descriptors that have been analyzed, the statistics of them.
So we see here the different statistics for spectrum complexity, etc.
All of the description data, pretty big list of all the low level features.
And again, these can be accessed directly from the API and
access specific items for that.
So that's pretty good.
So now, let's go into again into the resources and
we can then talk about the Client Libraries.
So the Client Libraries are the libraries that have been written
in a particular language, through which then we have access to this API,
and we have some code that facilitates this API call.
So let's go to this Python library.
Okay, so this is the library that has been written in Python to access the API.
So it's a simple library.
Basically, it has two major files.
One is this freesound.py that specifies a series of classes of functions,
through which we can do all these queries through the API.
So we can do the text search queries, or we can get sounds or
do combined search, or of different things.
And then there is Another file that is quite useful is the example file.
So in these examples, we have some example
pieces of code that perform different things.
So for example, they can perform, we request a sound,
for example here, and then we can just display the information about the sound or
we can get the analysis data, etc, etc.
Anyway, so let's use it so in order to use it the first
thing we will have to do is basically clone it so
we would just go here and get this ACPS URL and
then do from our computer git clone this and we'll get it.
But in order to use the API,
we need to have an API key.
Okay, so the first thing we need to do Is go
to a freesound and apply, so this is the,
again the developer's page of freesound and
we need to apply for a key, okay?
That's a key that is specific to us and
specific to our computer, and therefore it should be a personal.
So now I apply for a key and I will delete it after I do this demonstration.
But basically let's just use this key in this
lecture, to exemplify how to use it.
But you should apply for your own key and then use it.
And then you have this API key which should be secret, and you would use it.
Okay, so let's go to a text editor in which I copied
the files that this client library had.
It had this free sound.py file with all
these functions that are allows us to call API.
And then it has these examples file in which we can
do a number of calls to the API to put them as an example and
there was a place in which I have to put the API key.
So, in C-set token you would go to the where we
started before in here and in the examples,
And the reason of filing the examples
file this is where I have to type the API key and
so we go back to our text editor.
Here is where I placed the API key.
Now, in this example code, what I do is.
I get one instance of this free sound client.
This again is a object that I create, is an instance of an object is a class.
So this free sound client and then I have the variable C, and
now from this instance I can ask specific queries to the API.
So for example, I can get a given sound with a given ID and
this will return the sound.
In fact, this also, again an object.
And then I can query this object and for
example, obtain the low level analysis parameters.
And then print some of them.
Or I can ask for similar sounds, so I can get the list
of the most similar sounds based on these descriptors to this one.
And I can do other type of queries to these basic functions.
So for example instead of asking for a sounds, I can do a query of a text query
with some filters, and this will return these results.
And also, I can do other types of queries based on, for example, the content.
I can search for all the sounds that have a particular range of
a given level descriptor, for example, here a given
variance of pitch or a given salience, etc.
So let's, in fact, run this.
Let's go to terminal.
And I am in the directory where I have these files.
And I can just run examples.
.py, run examples.
Okay, now this is all of the results, it is pretty long, so
we are not going to be able to go through everything, but
the result of the examples it gives me the sound info, and
that is the first print statements that we had.
Then it gives me the analysis, the NFCCs that I ask.
It gives me similar sounds and names of those similar sounds.
Or then when I have this example of searching for
the word Monticello it gives me the names of different files.
And then again, that other search that is in here that
finds a specific queries to low level scriptures, etc.
And this last one is about the sounds of a particular user and
do something with that.
Okay so that's pretty good so
let's now go to one example that we are going to build.
So this is some code that in fact we wrote for
the assignment for the assignment of this week to facilitate you
to play around with the free sound API.
So, in this piece of code, what we are doing is facilitating the downloading
of a collection of sounds based on some specific query.
So we defined taking this freesound code from this client library.
We are putting together a function,
this downloadSoundsFreesound, that you can give query text.
Some tack, some duration, your API key and
then the directory that you want to put these sounds into and
then you can specify how many results it
should obtain and extension of the file.
And so what I did was to just write a little example that,
in fact, I could tie directly on Python,
that basically makes calls to this function.
So I import that file, the sound download file.
I typed my key here and these are three calls to this function.
So I want to get sounds from bassoon and
I want to store them in the tnp directory.
And I want to get 20, the top 20 results.
And with the restriction that duration has to be from 0 to 5 and
the tag has to be multisample and I do that for bassoon, guitar and
violin with kind of similar I guess, this bassoon I could put also from 0 to 3.
That will make it easier okay, so these are three function calls to this function.
Now, we're not going to go through that.
But basically, this is going to be part of your assignment.
And you will get this code.
And you will be able,
then, to write something like this to be able to get some sound.
So let's go to the terminal.
And again, here I have this get sounds file.
And if I run it, okay it obtains,
it downloads 20 sounds for every query.
This, of course, depends on the query labels.
These might not be possible if the tack is wrong, or the query's not correct,
we are not going to get so many sounds for that particular query.
We now go to the file system.
We are now in the tnp directory that I created.
This is the three folders that this little program created.
So I want to store the sounds of the bassoon, for example.
So these are 20 sounds and in every directory that
is the json file with the description of the low level
descriptors that I asked for, and the mp3 file.
So in fact the low level descriptions if we just open it with gedit.
Okay, I didn't ask for all of them, in fact if we go to sound
download again the only descriptions that have been asked for
are these ones, central contrast high frequency content,
MFCCs, lock at that time and inharmonisity.
So in my JSON file, we have all that, some of those files,
some of those descriptors like MSC has several values.
So this is the information we have for every sound.
I also ask for an mp3 file so we can listen to that to see if that works.
So now basically we have all these sounds,
all these directories with all the descriptions of the sound and
also sort of a little version of the sound so we can also listen to them.
And we can start doing things with that.
So, as part of the assignment, we are also distributing another Python code,
that facilitates doing the things
with this sound that we just obtained from the API.
So, as sound analysis go, you will get with the assignment
we have created different functions that
allows us to do display of these features and
do some K mains on the KNN type of processing.
That you will do in the lab, so no need for going through that now.
Let me just at least show you one of these functions.
This one, this is a very simple function.
What it does, it will display.
We give it this input directory,
the directory that we want to look for the sounds.
This is this TMP directory that I created.
And then I can specify the descriptors that I want to show.
And then it would plot it in a 2D plot, okay, of all the sounds that I have there.
So I have to figure out what I want to plot, so
this is all the descriptors that I have obtained.
So the number of descriptors was less, but
some of these descriptors had a number of different values for these.
So, for example, the MFCCs, so here we just have the mean of the first five.
Okay, so now I am in this the same directory.
I can Import the sound analysis file,
so that I have the functions here, while I can better import, for example, sa.
Okay, and now I can call sa dot, and
I can just get these function.
And I will ask for the directory is the tnp
directory where I have the sounds in, so this is the directory here.
And in terms of the descriptors, I can just put desk input and
I will, for example, use zero, which is a spectral centroid.
So here we see that zero is spectral centroid, and
maybe let's take one of the MFCCs, for
example, versus cc and that's number 12.
We are looking for 12 here and let's see.
That should do something, okay.
So here is the result.
I haven't paid much attention to what descriptive to use,
but in fact it's already interesting, and we can discuss
certain things about this plot.
The sounds I have are guitar sounds, violin sounds, and bassoon sounds.
Of course I have not listened to them, so I don't know exactly what they are like.
But well, it's interesting.
Bassoon, it's quite clustered.
So that means that all the sounds of the bassoon were quite similar.
Interestingly enough, the centroid is quite low, which makes sense.
Bassoon is a low instrument, and
the MFCC, this particular coefficient, number 1, is quite high.
The other two sounds, the violin and guitar, are quite mixed.
Maybe violin has a little bit higher MFCCs and a little bit lower centroid but
basically this central mean that the sounds are in the same register.
The violin in terms of the, yeah, it's a little bit more spread.
That's quite interesting and we can learn a few things about the sounds and
about the features guide trying to understand why
a particular set of features results in a particular distribution.
Of course, we have to listen to the sounds to make sure that
we understand what's going on.
And well, that's all I wanted to show in terms of the Freesound API.
Again, if you want to actually go into more information you should look
at the developer's page of Freesound and that should give you more information.
Okay, let's go back to here and
well, for references, again, Freesound.
And you need to apply for an API key so this is the URL for
applying for the API key, the documentation of the API.
And this client library that we are using.
So that is a Python wrapper to the API, so we have
access to the different resources of Freesound and through the API.
And finally we use two Python software
files that in fact come from assignment nine, from this week's assignment.
So if you want to reproduce what I just did,
you should look at the assignment and get that piece of code.
And that’s all.
So this was a presentation of the Freesound API.
Very much in tune with what we have been talking about this week which is to
analyze sounds and collections of sounds something like an API for
Freesound is basic in order to be able to do that in practice and
so forth, programming and exercising this
weeks type of work, we need such a tool.
And this is what we have presented.
So hopefully that was useful for you.
And sure it's not easy.
And there is a lot of things that
might have to learn if you want to really go deep into that.
But that gave you a basic introduction to it.
So thank you very much.
That's all and I hope to see you next class.
Bye bye.
