analog rhythm jam session

Thanks to Adrian Freed for drawing attention to this analog machine + human rhythm jam session “Viviendo la nochebuena en un ricon de sevillano aquí os muestro algo nuevo la máquina de compas y os deseo una feliz navidad a todos, con armonia, sentimiento y compas”

sensorimotor observations of collective movement, Heims Ingalls experiment

Hi Todd,

Thanks.  Can we talk with Pavan, and then Steve Heims?

Yes I’d be very interested in such collective movement experiments.  But then it is urgent that we really prep our own measurement methods and team (Garrett?, ___ ? assisted by Julian).

As you know, I would want to measure correlations not (only) in the brain but across much more of the event.   
It is far more direct (scientifically rigorous) to measure as much of the global aspects of collective movement than to zero in on only one part of the body and in fact a part whose functions are extremely indirectly related to corporal kinetics, and in ways that are quite ill understood . 

That’s why I’ve asked Julian and our students to build out the rhythm kit to use all modalities of sensing intervallic rhythm.

in particular: 

and as an aside:

Can we talk with Pavan, and then with Steve?

Xin Wei

On Dec 4, 2015, at 12:41 PM, Todd Ingalls <> wrote:

could this be tied to rhythm. I think we are both skeptical of brain imaging but could still be interesting. 

todd from my phone

Begin forwarded message:

From: Stephen Helms Tillery <>
Date: December 4, 2015 at 12:26:56 PM MST
To: Todd Ingalls <>
Subject: Back to music and brain

Hey Todd,

Hope you’re good.

I have been working and thinking about a couple of issues a lot lately in group neuroscience .. the two key topics are joint action and entrainment.   Joint action is just multiple actors working together to accomplish some task … like two people carrying a table together, or a couple of soccer players moving the ball down the field.   These are interesting problems because they require the actors to have some sense of what their partners are trying to accomplish and how they are going about that.   Entrainment is an entirely hypothesized process in which two brains come into “synchrony” in order to communicate .. this is thought to be important in language, but obviously is also important in music performance.

Entrainment, however, is pretty loosely defined at the moment … we have an idea for getting at entrainment using musicians.    The notion is to get an ensemble together, a good ensemble … and record simultaneous EEGs from the players as they work a piece.

To some extent this has been done before:   with saxophones (ugh!)   The focus of that paper was on EEG markers of empathy (even more ugh), and the usual expected changes in EEG associated with listening to and motor outputs for music.

What I’d like to do is do real analysis across multiple brains during performance, and see if we can see electrical signs of entrainment as they are working.   In a dream world, as the ensemble locks into “togetherness” … the brains will entrain.   Or vice versa.  

Anyway, to go after this we will need to synch up multiple EEGs, and more importantly, find a good ensemble that might be up for this.    

I thought of AME, and wondered if there would be somebody there interested in devoting a little bit of time and nominal resources to chasing this down.

In any case, have good holidays,


AME research and graduate proseminar: the problem with explaining things in terms of "'parts' of the brain"

Hardcastle and Stewart succinctly point out a fundamental problem at the heart of the methodology of neuroscience (and of cognitive science): the modularity thesis.

Neuroscience did not “discover” modules — loci of functions —  in brains.   Rather “they don’t even have a good way of accessing the appropriate evidence. It is a bias in neuroscience to localize and modularize brain functions.”

The problem with scientistic methodology is that you see what you expect to see.

There’s much more in play: Noah Brender’s work questions the modularity thesis underlying much of technoscience. 
However, another world is possible :)

Xin Wei

Synthesis rhythm: IMU's etc.

Dear Rhythm people: Garrett, Gabby, Julian,

Thanks for being on the demo team !  Now we can get back to steady state  work, like rhythmanalysis 

Can you please check out the IMU’s that we bought last year as an input for our rhythm test platform?
Ask Ozzie or perhaps one of Prof. Turaga’s students who’s used them for permission and see if you can stream them into Max.

I’d like assemble a suite of inputs:
contact mic
air mic
camera (Julian)
IMU (Pavan’s group?)
xOSC gyros (Mike —> Julian)

and record them in parallel
with some movement scenarios to get multiple streams of time data.

Please define some scenarios : e.g.  assembling blocks small to giant size, cutting and washing .  try seated and upper body and locomotive.   Varsha’s done some movement scenarios with Grisha, but in very specialzied contexts.  How about quotidian ?

Let’s try some out on Monday Nov 30?

Xin Wei

cc Pavan

Sha Xin Wei • Professor and Director • School of Arts, Media and Engineering + Synthesis
Herberger Institute for Design and the Arts + Fulton Schools of Engineering • ASU
Fellow: ASU-Santa Fe Consortium for Biosocial Complex Systems
Affiliate Professor: Future of Innovation in Society; Computer Science; English
Founding Director, Topological Media Lab
skype: shaxinwei • mobile: +1-650-815-9962

example of synthesis research: Naccarato and MacCallum, "From Representation to Relationality: Bodies, Biosensors, and Mediated Environments" JDSP 8.1 (2015)

Here’s a journal article published by a couple of researchers hosted at Synthesis last year that may be interesting to folks working on movement and responsive media, somatic experience, experimental dance and experimental technology, critical studies of technoscience, or philosophy of movement:

Teoma Naccarato, John MacCallum, “From Representation to Relationality: Bodies, Biosensors, and Mediated Environments,”  in Embodiment, Interactivity and Digital Performance, Journal of Dance and Somatic Practices, 8.1, 2015.

Teoma is starting a PhD with the Centre for Dance Research (C-DaRE), Coventry University UK
and John is a postdoc at the Centre for New Music and Audio Technologies (CNMAT) Department of Music, University of California at Berkeley. 

John and Teoma’s extended journal article is a good example of a durable outcome from the research cluster hosted by Synthesis in the Heartbeat Residency: Choreography and Composition of Internal Time.  This was a residency on temporality — sense of dynamic, change, rhythm — held February 15- 20, 2015. AME iStage, Matthews Center, ASU.

Ambient color changes according to whether dancer’s heart is faster or slower than some rate in the rhythm accompaniment software.  Synthesis Residency Jan 2015.   (The overhead tube lamps from Ziegler’s “forest2" were not used in this particular experiment.)

Improvisation with dancer Naccarato, composer / system creator MacCallum, Synthesis team and members of ASU laptop orchestra (Lorkas). Synthesis Residency Jan 2015.

Sha Xin Wei • Professor and Director • School of Arts, Media and Engineering + Synthesis
Herberger Institute for Design and the Arts + Fulton Schools of Engineering • ASU
Fellow: ASU-Santa Fe Center for Biosocial Complex Systems
Affiliate Professor: Future of Innovation in Society; Computer Science; English
Founding Director, Topological Media Lab
skype: shaxinwei • mobile: +1-650-815-9962

HMM in Max

On Fri, Apr 24, 2015 at 5:12 AM, Sha Xin Wei <> wrote:
Where can we get the best publicly available HMM external for Max
as a general purpose HMM package

Should we extend / modify  gf (which we have via IRCAM license )
and can we use it easily for non-audio data?   People claim to have tried it on video.
It seems that the real work is the preliminary feature extraction where a lot of interpretation happens
What are examples of code that do this in interesting ways?

Xin Wei

Navid Navab wrote 

While FTM is somewhat discontinued, this all is being moved to IRCAM's free Mubu package:
download the package and quickly check some of their example patches.


It contains optimized algorithms building on gf, FTM, cataRT, pipo, etc. While mubu is audio-centric, it is not necessarily audio-specific. mubu buffers can work with multiple data modalities and use a variety of correlation methods to move between these layers... This makes up a fairly wholesome platform without the need to move back and forth between gf, FTM, concatenative synthesis instruments, multimodal data handling, analysis, and etc.

As with most current IRCAM releases, it is highly under-documented. Besides gf that is distributed with their package, the mubu.hhmm object might be good place to start for what you are looking for:

also their xmm object might be of interest:

o4.track (video +sensor osc data), gyro (Was: Synthesis Center / Inertial Sensor Fusion)


Mike, Can you generate data in Julian’s data structure and store them in a shared directory for us all
along with the journaled video?  Julian Stein wrote the object for journalling data.

On Nov 10, 2014, at 1:07 AM, Julian Stein <> wrote:
Also included in the O4.rhyth_abs is a folder labeled o4.track. This features a simple system for recording and playing video with a synchronized osc data stream.
I’ll cc this to the SC team so they can point out those utilities on our github.

It’d be great if you can give the Signal Processing group some Real Live Data to matlab offline this week, as a warmup to Teoma + John’s data the week of Feb 15.

We must have video journaled as well, always.

I’d be interested in seeing an informal brownbag talk about Lyapunov exponents one of those mornings of the week of Feb 15, together with some analysis of the data. 

Let’s cc Adrian Freed and John MacCallum on this “gyro" thread —
Adrian’s got the most insight into this  and could help us make some actual scientific headway
toward publishable results.

My question is : By doing some stats on clouds of orientation measurements
can we get some measure of collective intention (coordinated attention)
not necessarily at any one instant of time (a meaningless notion in a relativistic world like ours) — 
but in some generalized (collective) specious present?

Let’s plan John and Teoma’s workshop hour by hour schedule this coming week at a tea?

Kristi or Garrett, or __: please let us know when the “heartbeat”  workshop weebly site is posted and linked to the Synthesis research ok?

Xin Wei

On Feb 6, 2015, at 12:13 PM, Michael Krzyzaniak < wrote:

I translated Seb's sensor fusion algorithm into Javascript to be used within Max/MSP:

There was still quite a bit of drift when I tested it, but I was only using 100Hz sample rate which I suspect may have been the main issue.


 On Sat, Jan 31, 2015 at 3:45 PM, Adrian Freed <
  Thanks Xin Wei.
  Tt would indeed to at least develop a road map for this important work. We should bring the folk from x-io
  into the discussion because they have moved  their considerable energies and skills further into this space in 2015.
  I also want to clarify my relative silence on this front. As well as weathering some perfect storms last year, I found
  the following project attractive from the perspective of separating concerns for this orientation work:
  They are still unfortnuately in pre-order land with a 2-3 month shipping time. Such a system would complement commercial and inertial measuring systems well
  by providing a "ground truth" ("ground fib") anchored to their beacon transmitter.  The sixense system has limited range for many of our applications
  which brings up the question (again as a separation of concerns not for limiting our perspectives) of scale. Many folk are thinking about orientation and inertial
  sensing for each digit of the hand (via rings).
  For the meeting we should prepare to share something about our favored use scenarios.

 On Jan 31, 2015, at 1:37 PM, Xin Wei Sha <
  Can you — Adrian and Mike —  Doodle a Skype to talk about who should do what when to get gyro information from multiple (parts of ) bodies
  into our Max platform so Mike and the signal processing maths folks can look at the data?
  This Skype should include at least one of our signal processing  Phd’s as well ?
  Mike can talk what he’s doing here, and get your advice on how we should proceedL
  write our own gyro (orientation) feature accumulator
  get pre-alpha version of xOSC hw + sw from Seb Madgewick that incorporates that data
  adapt something from the odot  package that we can use now.
  WAIT till orientation data can be integrated easily (when, 2015 ?)
  Half an hour should suffice.
  I don’t have to be at this Skype as long as there’s a precise outcome and productive decision that’ll lead us to computing some (cor)relations on streams of orientations as a start...
  Xin Wei

On Jan 31, 2015, at 1:27 PM, Vangelis < wrote:

  Yes, there is great demand for something that works in sensor fusion for inertial sensors but I think the best way to do so is as part of o. so to benefit every inertial setup out there. It will take ages for Seb to implement it for xosc and would be an exclusive benefit. Seb's PhD is out there and I am sure he will help sharing new code for solving the problem. The question is can we do this? :)
  My warm regards to everyone!

On Jan 30, 2015 6:45 PM, Adrian Freed <

  The experts on your question work at x-io. Seb Madgewick wrote the code a lot of people around the world are using for sensor
  fusion in IMU's.
  Are you using their IMU (x-OSC) as a source of inertial data?
  We started to integrate Seb's code into Max/MSP but concluded it would be better to wait for Seb
  to build it into x-OSC itself. There are some important reasons that this is a better approach, e.g.,
  reasoning about sensor fusion in a context with packet loss is difficult.
  It is possible Vangelis persisted with the Max/MSP route
  On Jan 30, 2015, at 3:01 PM, Michael Krzyzaniak <  wrote:
  Hi Adrian,
  I am a PhD student at ASU and I work with Xin Wei at the Synthesis Center. We are interested in fusing inertial sensor data (accel/gyro/mag) to give us reliable orientation (and possibly position) information. Do you have in implementation of such an algorithm that we can use in (or port to) Max/MSP?