posted 26 April 2014 19:18            

using the phases of n frames to split the spectrum into noise and sinusoidal components. depending on how much the unwrapped phases are changing (increase), it should be possible to estimate which freq-bins content is "noisey", as they changing steadily , right?

beside the possibility to surpress noise or sinusoidals in a signal it could also been used to pitch the sinusoids without touching the noise etc.

your feedback is very welcome.

thanks,
jo

posted 27 April 2014 02:23            

Yes this is something I've been working on for a long time. The intention is that the noise and the tonal get routed to two different oscillator banks and get re-synthesized separately then mixed. In theory this should give a much better speech re-synthesis and most non polyphonic signals should be improved.

Did you see the "wire between" video where I spoke about the problems of re-synthesizing signals that were a mix of both tonal and non tonal sound and how it became comb filtered and sounded metallic, well this is probably the way to cure it, but it is not as easy as it may first sound. That said with a lot of research and experimentations it could be achieved.

Things to consider.
The tonal parts can be found and reproduced by first smoothing and subtracting the the smoothed from the un-smooth and adjusting the amplitudes to with the amount of residue. The residue also needs to be smoothed before adjusting else it will add noise to the tonal. The problem here is to do the adjusting as you need a sample rate divide module to proportion the amplitudes. There was a sample rate divide module in Petes DSP modules but that doesn't work on the Paca, but I suspect/hope SSC may bring out one soon. In the mean time the arctan module can be used as a sort of divide or there may be other get rounds. Also vibrato would be part of this residue and we would want that to be taken away from the tonal path so combined smoothers can act as a lo frequency band pass filter.

Also we need to think about the noise as transient provider as well. So with all the smoothers used for the detection we must keep the fast changing signals in tack. Also the Noise/transient Osc bank needs to always be fed with just the right amount frequency and amplitude jitter to avoid the comb filtering effect.

Also timing adjustments will be needed to make sure that the two parts join back together correctly.

This is all very hard but I believe it's posable.

posted 27 April 2014 14:55            

i know that this is a hard nut to crack. axel röbel and others at ircam´s analysis synthesis team worked on this for a long time and developed a great phasevocoder kernel that is part of audiosculpt and a bunch of "realtime" resynthesis max objects.

just like you mentioned, this pvoc takes care of the transients and enables the mixing of it together with noisy and sinusoidal components of the spectrum. but of course it is not modular. this is the reason i would love to use the noise/sinusoid components splitting as a "extra high quality sound" in kyma.

"The tonal parts can be found and reproduced by first smoothing and subtracting the the smoothed from the un-smooth and adjusting the amplitudes to with the amount of residue."
what exactly you meant to smooth? the amps?

and wouldnt we want to keep vibrato (pulsating change of pitch) in the tonal part?

and did you also experimented with the phase (as i described in my first post)...
do you think it could lead to a "extra high quality sound"?

thanks for all the hints. i will watch the part of your wire in between-video tonight.
greetings, j

posted 27 April 2014 15:35            

By smoothing the amps I mean the same thing we have been doing by splitting the spectrum in ram into amps and freqs and using the delay line with feedback to smooth them.

With the spectrum in ram we get freqs, not phases, so it makes life easier. The noise can be found as changes in freq and amp from frame to frame. Changes in freq is similar to changes in phase except that for non noisy signals the phases will change cyclicly if the incoming frequency is not in the centre of the band. The spectrum in ram frequencies have already accounted for this and so it is one less thing to worry about.

posted 30 October 2014 14:05            

Imagine you took partial one average level of 0.5 and the average or smoothed peaks of the wiggles (either rectified or squared) came to 0.1 then in this case you would subtract the 0.1 from the 0.5 to get the smoothed level. Now if instead you wanted to use the frequency wiggle as the decider. By experiment you choose a max wiggle % amount as ref for all noise, (i.e frequency of 1khz wiggling by 10 hz would equate to the same value as 100hz wiggling by 1 hz. Log pitch would be more direct. If you use that value, then the % of the max wiggle would be used to proportion the amp level to either noise output or tone output.

Needs more thinking about and experimentation to fine the best way to go. maybe a combination of amp wiggle and freq wiggle will give the best control signal. Also not the the noise control will need to be faster than the tone control so may be it would be better to leave the rectified wiggle residue un smoothed as the partials noise control.

posted 31 October 2014 03:29            

I know we already talked about this but it's difficult to grasp By "average" you mean the arithmetic mean? Like the sum of the amplitudes of 5 frames divided by 5? Then I wonder why I would get the smoothed level if I subtract the average wiggle from the average level. Isn't smoothing already some kind of running average?

What do you mean by "recitified or squared"?

I thought you just subtract the smoothed amps from the unsmoothed amps. There I get the amp wiggle (nice non-scientific word BTW ). Now I use the % of the max wiggle (derived from the frequencies) to proportion the amount of amp wiggle in each output (tonal/noise). I see only one problem here: When one partial is used to produce noise the frequency is wiggling very fast but the amplitude still could be quite constant resulting in a very low amp wiggle. Or is a fast frequency wiggle always accompanied by a high amp wiggle?

Thanks,
Gustl

posted 03 November 2014 12:34            

Wouldn't it be a good idea to convert the amps to log scale db? Then the AmpDifferences will be accurate no matter how high/low the average amplitude is.. What I mean is a wiggle of 0.1 is a lot when the smoothed is 0.2 but not that much if the smoothed is 0.8. On the other hand if we divide the wiggle by the amplitude level we get proportions which seems right as well.

About the frequency wiggle: It's not that easy to track the frequency of the wiggle of each partial. I came up with this: Use a threshold module on the frequency wiggles with threshold 0. This should give a trigger each time the wiggle goes from minus something above 0. So each period we should get a trigger. Now multiply that signal by 0.01 and sum up the values of consecutive frames by delaying them (1st frame: 20 frames delay, 2nd frame: 19 frames delay, and so on) and sum the values in a mixer. If we do that for 0.1 s (about 20 frames) we get the amount of periods of the wiggle for 0.1 s ( e.g. 0.05 would mean 5 periods). Multiply by 10 if needed to get the frequency (periods per second). Will try this but maybe there is an easier way?
The weighting of the resulting wiggle frequency could be done by multiplying them with a log function, but I haven't thought about that yet.

posted 10 November 2014 02:38            

RunningSum9s.kym

It works as described and I've attached an example. To get the running sum (or number of triggers) you just use a delay and subtract the triggers after a certain time. In the example I used the threshold module and differentiated it, then got rid of one trigger (otherwise I would get 2 triggers, one for threshold exceeded, one for going back to 0). I did this to get a trigger which is 1 sample long. In the example I'm sending a trigger every second and set the running time to 9 s. So it counts till 9 and stays there. The slow triggering is just for demonstration, you can actually count at sample rate with this. And you can do summing because that's what it actually does
What are you going to do with it?

posted 13 November 2014 13:18            

I was interchanging the word average and smooth but meaning the same thing. what I was suggesting was that (as you said) subtracting the smoothed amps from the un-smoothed amps, but this will give a wiggle that would go both plus and minus. If there is lots of wiggle we would expect the pitched part of the sound to be low and the noise part of the sound to be high. So we need a measure of wiggle amount to know how much to reduce the smoothed amps. If we just averaged (smoothed) the wiggle, because it is both plus and minus it would end up as nothing so we need to either rectify (absolute value) or square it before smoothing to get a value that represents how noisey the partial is. Then this value is subtracted from the smooth to give that new amps that feed ocs bank bank one.

The same rectified smoothed value will feed the amps second oscillator bank to produce the noisy part of the signal separately. Of course the first osc bank would need the pitches smoothed to keep it clean and the second osc bank would use the same clean pitches but have noise (wiggles) added, and the amps would also have noise (wiggles) added to make this bank generate only noise. These two wiggles would be best taken from analyzed white noise querked (ie with the smoothed portion subtracted from them) to make sure that this oscillator bank gives just the right amount of wiggle in both pitch and amplitude to produce nicely spectrum filled noise and not sound like a comb filtered version. This may be hard to achieve when you still want control over the amplitude level and general pitch centering, so may be (as you say) it should be done using log amplitudes or multiply instead of adding. Not yet sure about this.

I don't see the benefit in counting how many times the wiggle crosses zero if this is what you were thinking more just how loud the wiggle is.

does this make sense?

posted 13 November 2014 14:16            

As I see it, it is completely impossible for us to hear any phase difference between harmonics. A good test is to put a very complex sound into the hilbert transform which completely screws up the relationship between the harmonic phase and yet you cannot tell the difference at all. You can even put it through cascades of all pass filters and compare it with the original and there is no audible difference what so ever unless you have so many modules that you start introducing different delays at different frequencies in the signal.

Note however that you can tell the difference between phases of signals fed to your left and right ears, but only for frequencies below about 1 khz.

But maintaining the phase relationship in analysis does improve things, but for a very different reason. As some components of the sound may sit in more than one band (especially signals on the border between bands) it gets reconstructed by both bands and can subtract or add if the phase relationship between those bands are not maintained. This will alter the spectral content almost randomly and erratically and this is very audible.

Being aware if this is probably one of the most important aspects to doing good analysis and spectral manipulation. If the bands in your analysis moved with pitch of the signal and kept centered on the harmonics of the signal, you would have no any problems with phase deviations except for the noise/breathy components which lay between the harmonics that give the a metallic lo fi phasey sound.

This is why if you can successfully strip out the noise and tonal components separately from a signals that has both at the same time (not as in the video switching back and forth) you should get a far superior analysis without getting hung up on the harmonic phase relationship which we can't hear.

posted 13 November 2014 15:47            

Thanks for posting.

posted 15 November 2014 18:55            

NewSRLogic.kym

Attached are some basic sample rate logic building bricks that could be used to do the trigger counter. They will probably need to be encapsulated first remembering that some contain feedback modules which will need unique feedback names and need to be encapsulated with the name in it's own field.

The basic idea is you have the zero crossing pulse feeding a flipflop feeding a flipflop feeding a flipflop and the third has a pair of feedback modules to reset the first and second. And the first resets the third.

hope it helps

posted 16 November 2014 12:54            

Interesting but how are you going to record the microsounds? The MemoryWriter can't be triggered at samplerate.. I'm interested in this for implementing the psola algorithm in kyma (http://en.m.wikipedia.org/wiki/PSOLA), so please let me know, thanks!

Hi Pete,

Alright, now it's clear I think it's better to calculate the amount of wiggle by dividing the absolute wiggles by the smoothed amps and then multiply the result with the amps for the noise and multiply (1 - the result) for the tonal. this way you keep the ratio right. I'm still experimenting though.

I was counting the frequency wiggles' zero crossing because I was assuming that if the frequency is changing fast it is producing noise. But it seems that either my patch is wrong or it is not the case.. I also have some good results when I treat the frequencies like the amps and measure the amount of wiggle. The whole thing reminds me of tracking formants: either it's too slow and smears or it's not exact enough or giving wrong results - it's a pity.

Another thought on this: When we do a harmonic analysis we should be able to track the sinusoidal parts (the harmonics) better. So maybe it makes sense if you subtract the harmonic analysis from the non-harmonic to get the the residual (noise). I have to try..

Totally agree about the phase relationship. The quality of the tonal-noise split depends on the quality of tracking of the harmonics. This is why I think the harmonic analysis can serve our needs even better. Actually it should already know which parts are tonal and which are noise because the algorithm decides which partial to track, where is a jump, what is noise, what is a harmonic of the fundamental. So maybe there's a way for SSC to build a tonal-noise module which would be superior than our ideas? Just thinking loud, though

Thanks for the SR Logic stuff, will check it out tomorrow!

posted 26 December 2014 18:55            

Regarding the splitting of the noise and tonal parts, it's difficult to know if your getting close or what values to put for smoothing etc. One way is to start with noise and tone test signals and monitor them as separate signals (switchable) with separate level controls. Then mix the two together and analyze them and put it through your separator. Then compare the separate resynthed outputs with the separate direct inputs and see how they compare level wise and sound wise. Try filtered noise on the test signals and see how that compares. Then try moving tones and switching the tone and noise on and off and see how the separator copes. This way there is a goal to work towards and a way of telling if it's working as it should.

hope this makes sense

posted 07 January 2015 07:23            

That's a good idea to make this kind of test setup! I've put further development on cold storage for now but I will continue working on this some time. Anyway, I found some really nice Sounds Some of them you can hear for the Vocal Design of a german movie which has not been released yet. I'll do a tutorial on this once everything is official

posted 10 November 2015 13:54            

Kyma's spectral analysis is a sinusoidal model and therefore there must be some sort of algorithm tracking the peaks in the frequency domain and derive their true amplitude and frequency (maybe through parabolic interpolation?). So that's the data we are getting..

What if we run another sinusoidal model on this? This could be done by defining the following parameters (most of them need to work weighted by frequency BTW):

- amplitudeDeviation: setting the range of allowed amplitude deviation over time
- frequencyDeviation: setting the range of allowed frequency deviation over time
- minDuration: if a partial passes the tests there should be a minimum duration considered as final test

If a partial passes all the tests it can be considered as stable (which may be tonal). Those partials get subtracted from the original partials to get the residual which can be considered as unstable (which might be noise). Or one can subtract the resynthesised partials from the original signal in the time domain, which might be even better.
All this involves quite some delay, at least minDuration s. Also it's crucial to set the time values for all 3 parameters right - and don't forget frequency weighting. Lots of work!

posted 10 November 2015 14:33            

The pitched set would take the smoothed partial pitches as it's frequencies, and the derived pitch energy as it's amplitudes.

The noise leg would take an analysed white noise. The frequencies would go through un altered but the amplitudes would be the amplitudes multiplied by the noise energies calculated before .

I don't know if it will work, but just may.

posted 10 November 2015 16:06            

As for the other factors: I hope SSC did their best to minimize those kind of problems, I can only work with the data I get. Unless I would do my own sinusoidal model using the FFT (which would be very inefficient BTW).

posted 10 November 2015 17:54            

Subtracting the tone from the original means that you have to keep phase information which is very hard when it's amplitude and frequency type data and not real and imaginary. But even if you did your own FFT it would still need converting to frequ and amp so you don't gain anything.

You are in effect subtracting the tonal data info by using the splitting method I described , I hope.

posted 11 November 2015 01:23            

I see what you mean about the phase, you're right..

posted 03 December 2015 03:04            

it works within a wire-in-between network, right?
any soundcloud demo yet… so we get an idea how it sounds?