Affiliate Professor: Future of Innovation in Society; Computer Science; English
Founding Director, Topological Media Lab
Where can we get the best publicly available HMM external for Max
as a general purpose HMM package
Should we extend / modify gf (which we have via IRCAM license )
and can we use it easily for non-audio data? People claim to have tried it on video.
It seems that the real work is the preliminary feature extraction where a lot of interpretation happens
What are examples of code that do this in interesting ways?
Xin Wei
Navid Navab wrote
Also included in the O4.rhyth_abs is a folder labeled o4.track. This features a simple system for recording and playing video with a synchronized osc data stream.
I translated Seb's sensor fusion algorithm into Javascript to be used within Max/MSP:
There was still quite a bit of drift when I tested it, but I was only using 100Hz sample rate which I suspect may have been the main issue.Mike
On Sat, Jan 31, 2015 at 3:45 PM, Adrian Freed <adrian@adrianfreed.com
wrote:
Thanks Xin Wei.
Tt would indeed to at least develop a road map for this important work. We should bring the folk from x-io
into the discussion because they have moved their considerable energies and skills further into this space in 2015.
I also want to clarify my relative silence on this front. As well as weathering some perfect storms last year, I found
the following project attractive from the perspective of separating concerns for this orientation work: http://store.sixense.com/collections/stem
They are still unfortnuately in pre-order land with a 2-3 month shipping time. Such a system would complement commercial and inertial measuring systems well
by providing a "ground truth" ("ground fib") anchored to their beacon transmitter. The sixense system has limited range for many of our applications
which brings up the question (again as a separation of concerns not for limiting our perspectives) of scale. Many folk are thinking about orientation and inertial
sensing for each digit of the hand (via rings).
For the meeting we should prepare to share something about our favored use scenarios.
On Jan 31, 2015, at 1:37 PM, Xin Wei Sha <Xinwei.Sha@asu.edu
wrote:
Can you — Adrian and Mike — Doodle a Skype to talk about who should do what when to get gyro information from multiple (parts of ) bodies
into our Max platform so Mike and the signal processing maths folks can look at the data?
This Skype should include at least one of our signal processing Phd’s as well ?
Mike can talk what he’s doing here, and get your advice on how we should proceedL
write our own gyro (orientation) feature accumulator
get pre-alpha version of xOSC hw + sw from Seb Madgewick that incorporates that data
adapt something from the odot package that we can use now.
WAIT till orientation data can be integrated easily (when, 2015 ?)
Half an hour should suffice.
I don’t have to be at this Skype as long as there’s a precise outcome and productive decision that’ll lead us to computing some (cor)relations on streams of orientations as a start...
Cheers,
Xin Wei
__________
Hello!
Yes, there is great demand for something that works in sensor fusion for inertial sensors but I think the best way to do so is as part of o. so to benefit every inertial setup out there. It will take ages for Seb to implement it for xosc and would be an exclusive benefit. Seb's PhD is out there and I am sure he will help sharing new code for solving the problem. The question is can we do this? :)
My warm regards to everyone!
v
On Jan 30, 2015 6:45 PM, Adrian Freed <adrian@adrianfreed.com
wrote:
Hi.
The experts on your question work at x-io. Seb Madgewick wrote the code a lot of people around the world are using for sensor
fusion in IMU's.
Are you using their IMU (x-OSC) as a source of inertial data?
We started to integrate Seb's code into Max/MSP but concluded it would be better to wait for Seb
to build it into x-OSC itself. There are some important reasons that this is a better approach, e.g.,
reasoning about sensor fusion in a context with packet loss is difficult.
It is possible Vangelis persisted with the Max/MSP route
On Jan 30, 2015, at 3:01 PM, Michael Krzyzaniak <mkrzyzan@asu.edu wrote:
Hi Adrian,
I am a PhD student at ASU and I work with Xin Wei at the Synthesis Center. We are interested in fusing inertial sensor data (accel/gyro/mag) to give us reliable orientation (and possibly position) information. Do you have in implementation of such an algorithm that we can use in (or port to) Max/MSP?
Thanks,
Mike
Not only handcraft manufacture, not only artistic and poetical bringing into appearance and concrete imagery, is a bringing-forth, poiesis. Physis also, the arising of something from out of itself, is a bringing-forth, poiesis. Physis is indeed poiesis in the highest sense. For what presences by means of physis has the bursting open belonging to bringing-forth, e.g., the bursting of a blossom into bloom, in itself (en heautoi). In contrast, what is brought forth by the artisan or the artist, e.g. the silver chalice, has the bursting open belonging to bringing forth not in itself, but in another (en alloi), in the craftsman or artist.
[Heidegger, Question Concerning Technology, 11]
From: Adrian Freed <adrian@cnmat.berkeley.edu>
Subject: Re: Automatic projector calibration
Date: October 5, 2014 at 1:08:24 PM MST
The technical notion of lag, does not jive very well with the multiple temporal structures involved in experience.
Using it as a ground truth produces some ugly theories, e.g., http://en.wikipedia.org/wiki/Interaural_time_difference
Notice the frequency dependent hacks added to the theory and vagueness about delay/phase. Also notice that the detailed
anatomical and signal flow analysis says nothing to support the theory other than that there are places the information from each ears meet. I encourage everyone to think this through carefully and build and explore speculatively.
We have been down this hole at CNMAT for pitch detection on guitars. People think you can synthesis sounds that tightly track pitch of a string. You can't. There are no interesting definitions of a low guitar string pitch that would make this possible. One "solution" is to track with a constant lag, i.e. a sort of echo. This conditions and constrains the space considerably. A few artists have done amazing things within these constraints (https://www.youtube.com/watch?v=VYCG5wZ9op8, https://www.youtube.com/watch?v=1X5qDeK3siw) but the apparatus has strong agency and may well interfer with other goals.
On Oct 5, 2014, at 10:44 AM, Evan Montpellier <evan.montpellier@gmail.com> wrote:Pages 36-9 in the thesis deal with tracking moving projection surfaces
at near real-time rates. The short of it is that the max refresh rate
Dr. Lee was able to achieve was 12Hz, noting:
"feedback latency places a substantial constraint on the usage of
alternative patterns that may utilize recent sensor data to improve
tracking performance. Tracking algorithms that require instantaneous or
near instantaneous feedback from sensors are not likely to be executable
in practice."
Perhaps the lag would be be acceptable, though, within some of the
visual movement experiments that already play with time delay.
Evan
On 2014-09-30, 4:47 PM, Byron Lahey wrote:From my perspective, the real value that would come from implementing an
auto-calibration system would be the potential for dynamic projection
surfaces: surfaces that enter or exit a space, expand and contract,
morph into different shapes, etc.
I'm interested but don't have much bandwidth to devote to this.
Byron
On Fri, Sep 26, 2014 at 1:36 PM, Evan Montpellier
<Evan.Montpellier@asu.edu <mailto:Evan.Montpellier@asu.edu>> wrote:
For projects such as the Table of Content/Portals, automatic
projector calibration would save a considerable amount of work and
time. Here's an attractive looking solution from Dr. Johnny Chung
Lee, presently of Microsoft:
http://johnnylee.net/projects/thesis/
Is anyone interested in attempting to implement an analogous system
as part of the Synthesis-TML portal network?
From: Xin Wei Sha <Xinwei.Sha@asu.edu>
Subject: Re: [Synthesis] Notes for Lighting and Rhythm Residency Nov (13)17-26
Date: October 4, 2014 at 2:31:10 PM MST
Please please please before we dive into more gear specsWhat are the experiential provocations being proposed?For example, Omar, everyone, can you please write into some common space more example micro-studies similar to Adrian’s examples?(See the movement exercises that MM has drawn up for past experiments for more examples.)Here at Synthesis, I must insist on this practice, prior to buying gear, so that we have a much greater ratio ofpropositions : gadgets.Thank you, let’s play.Xin Wei
_________________________________________________________________________________________________
On Oct 4, 2014, at 12:53 PM, Omar Faleh <omar@morscad.com> wrote:I got the chance lately to work with the Philips Nitro strobes which is intensely stronger than the atomics 3000 for example. It is an LED strobe, so you can pulse, flicker, and keep on for quite a while without having to worry about discharge and re-charge.. and being an all-led strobe, it isn't as voltage- hungry as the atomics..The LED surface is split into 6 sub rectangle that you can address individually or can animate by preset effects, which allows for a nice play with shadows with only one light (all DMX-controlled)
and there is an RGB version of it too.. so no need for gels and colour changers.I am also looking into some individually-addressable RGB LED strips . Placing the order today so I will hopefully be able to test and report the findings soon
_________________________________________________________________________________________________
On 2014-10-04, at 3:30 PM, Adrian Freed <adrian@adrianfreed.com> wrote:Sounds like a fun event!
Does the gear support simple temporal displacement modulations, e.g., delaying one's shadow or a projected image of oneself?
This is rather easy to do with the right gear.
I would like to see something more ambitious attempted along the lines of things we have played with using sound. By tracking feet we can produce the anticipatory sound of a foot fall which messes with the neat and tidy
notion of retentions and protentions. Can you rework a visual representation of oneself (shadow, image, silouette, ghost) to move in anticipation of where one moves?
It would also be interesting to modulate a scrambling of oneself and connect its intensity to movement intensity. Navid has done things similar to this with sound. The experience was rather predictible but might well be different visually.
_________________________________________________________________________________________________
On Oct 4, 2014, at 11:53 AM, Xin Wei Sha <Xinwei.Sha@asu.edu> wrote:Chris, Garrett, Julian, Omar, Chris Z, Evan, Byron, Prashan, Althea, Mike B, Ian S, Aniket, et al.
This is a preliminary note to sound out who is down for what for the coming residency on Lighting and Rhythm (LRR).
The goal is to continue work on temporality from the IER last Feb March, and this time really seriously experimentally mucking with your sense of time by modulating lighting or your vision as you physically move. First-person experience, NOT designing for spectator.
We need to identify a more rigorous scientific direction for this residency. Having been asking people for ideas — I’ll go ahead and decide soon!
Please think carefully about:
Core Questions to extend: http://improvisationalenvironments.weebly.com/about.html
Playing around with lights: https://vimeo.com/tml/videos/search:light/sort:date
Key Background: http://textures.posthaven.comThe idea is to invite Chris and his students to work [richly] on site in the iStage and have those of us who are hacking time via lighting play in parallel with Chris. Pavan & students and interested scientist/engineers should be explicitly invited to kibbutz.
• Lighting and Rhythm
The way things are shaping up — we are gathering some gadgets to prepare for .
Equipment requested (some already installed thanks to Pete Ozzie and TML)
Ozone media system in iStage
Chris Ziegler’s Wald Forest system (MUST be able to lift off out of way as necessary within minutes — can an inexpensive motorized solution be installed ?)
3 x 6 ? grid of light fixtures with RGB gels, beaming onto floor
IR illuminators and IR-pass camera for tracking
Robe Robin MiniMe Moving Light/ Projector
Hazer (?)
Strobe + diffuser (bounce?)
+ Oculus DK1, (Mike K knows )
+ Google Glass (Chris R can ask Cooper , Ruth @ CSI)
We need to make sure we have a few rich instruments (NOT one-off hacked tableaux!) coded up ahead of time -- hence the call to Max-literate students who would like to try out what we have in order to adapt them for playing in the LRR by November.
Note 1:
Let’s be sure to enable multiplex of iStage to permit two other groups:
• Video portal - windows : Prashan, Althea Pergakis, Jen Weiler
• shadow puppetting, Prashan working with Byron
Note 2:
Garth’s Singing Bowls are there. Think about how to integrate such field effects.
Mike can you provide a Max patch to control them — ideally OSC -- but at least to fade up/down without having to physically touch any of the SB hardware.
Note 3:
This info should go on the lightingrhythm.weebly.com experiment website that the LRR leads should create Monday unless someone has a better solution — it must be editable by the researchers and experiment leads themselves. Clone from http://improvisationalenvironments.weebly.com !
Xin Wei
_________________________________________________________________________________________________
On Sep 4, 2014, at 8:53 AM, Adrian Freed <adrian@adrianfreed.com> wrote:
I am afraid I can't be very helpful here. I don't do MIR work myself. The field for the most part does
offline analyses of large data sets using musicologically naive Western musical concepts of pitch and rhythm.
One exception to the realtime/offline choice is from our most recent graduate student to work on the
beat tracking problem, Eric Battenburg. Here is his dissertation: http://escholarship.org/uc/item/6jf2g52n#page-3
There is interesting machine learning going on in that work but it presumes that one can make a reliable
onset detector which is a reasonable (but narrow) assumption for certain percussion sounds and drumming practice.
The questions of phase and "in sync." raised below interest me greatly. There are is no ground truth to the beat
(up or down or on the "beat"). I remember being shocked recently to discover that a bunch of research on dance/music entrainment relied as a reference on hand-labeled visual beat markings from "expert listeners in the computer music lab next door" . Various concepts such as "perceptual onset time" have been developed to sufficiently complicate this question and explain the difficulty people have observing concensus on musical event timing and relating a particular beat measurement to features of the acoustic signals.
Even a "simple" case, bass and drums, is extremely difficult to unravel. The bass being a low frequency instrument complicates the question of "onset" or moment of the beat. The issue of who in this pair is determining the tempo
is challenging and the usual handwaving that the tempo is an emergent coproduction of the performers is not very helpful in itself in elaborating the process or identifying which features of the action and sound are relevant to the entrainment. My guess is that we will find models like the co-orbital arrangment of Saturn's moons Epimetheus and Janus.
What are the system identification tools to reveal these sorts of entrainment structures? Can this be done from the sound
alone or do we have to model embodied motions that produce the sounds?
NOTE from Adrian XW Mike Krzyzaniak on Percival-Tzanetakis Tempo Estimator :
_________________________________________________________________________________________________
On Sep 3, 2014, at 6:38 AM, Sha Xin Wei <shaxinwei@gmail.com> wrote:Phase: I’m interested in both the convention of syncing on peaks
but also in the larger range of temporal entrainment phenomena that Adrian has identified with suggestive terminology.
In practice, I would apply several different measures in parallel.
Yes, it would be great to have a different measure. For example, one that detects when a moderate number (dozens to 100) of irregular rhythms have larger number of simultaneous peaks. This is a weaker criterion than being in phase, and does not require periodicity.
Xin Wei
_________________________________________________________________________________________________
On Sep 2, 2014, at 5:07 PM, Michael Krzyzaniak <mkrzyzan@asu.edu> wrote:Of course we could reduce the 6 second lag by reducing the window sizes and increasing the hop sizes, at the expense of resolution. Also, rather than using the OSS calculation provided, perhaps we could perhaps just use a standard amplitude follower that sums the absolute value of the signal with the absolute value of the Hilbert Transform of the signal and filtering the result. This would save us from decimating the signal on input and reduce the amount of time needed to gather enough samples for autocorrelation (at the expense of accuracy, particularly for slow tempi).
What are you ultimately using this algorithm for? Percival-Tzanetakis also doesn't keep track of phase. If you plan on using it to take some measure of metaphorical rhythm between, say, humans as they interact with each other or the environment, then it seems like phase would be highly important. Are we in sync or syncopated? Am I on your upbeats or do we together make a flam on the downbeats?
Mike
_________________________________________________________________________________________________
On Tue, Sep 2, 2014 at 4:09 PM, Sha Xin Wei <shaxinwei@gmail.com> wrote:
Hi Adrian,
Mike pointed out what for me is a serious constraint in the Percival-Tzanetakis tempo estimator : it is not realtime.
I wonder if you have any suggestion on how to modify the algorithm to run more “realtime” with less buffering if that’s the right word for it…
Anyway I’d trust Mike to talk with you since this is more your than my competence. cc me for my edification and interest!
Xin Wei
_________________________________________________________________________________________________
On Sep 2, 2014, at 12:06 PM, Michael Krzyzaniak <mkrzyzan@asu.edu> wrote:Hi Xin Wei,
I read the paper last night and downloaded the Marsyas source, but only the MATLAB implememtation is there. I can work on getting the c++ version and porting it, but the algorithm has some serious caveats that I want to run by you before I get my hands too dirty.
The main caveat is that it was not intended to run in real-time. The implementations they provide take an audio file, process the whole thing, and spit back one number representing the overall tempo.
"our algorithm is more accurate when these estimates are accumulated for an entire audio track"
It could be adapted to run in sort-of real time, but at 44.1k the tempo estimation will always lag by 6 seconds, and at a control rate of 30ms (i.e. the rate touchOSC uses to send accelerometer data from iPhone) the algorithm as described will have to gather data for over 2 hours to make an initial tempo estimation and will only update once every 5 minutes.
Once I get the c++ source I can give an estimation of how difficult it might be to adapt (in the worst-case scenario it would be time-consuming but not terribly difficult to re-implement the whole thing in your language of choice).
If you would still like me to proceed let me know and I will contact the authors about the source.
Mike
________________________________________________________________________________________________
On Mon, Sep 1, 2014 at 3:45 PM, Sha Xin Wei <shaxinwei@gmail.com> wrote:
beat~ hasn't worked well for our research purposes so I'm looking for a better instrument.
I'm no expert but P & T carefully analyze the extant techniques.
the keyword is 'streamlined'
Read the paper. Ask Adrian and John.
Xin Wei