About signed vs. unsigned position values, and left border resizing, and stuff

[terminator356]

Hiya! Should have a commit soon of some fixes, improvements,
mostly regarding wave editor and waves in the arranger.

@spamatica 's work on border resizing, especially the left border, especially with wave parts,
needed a bit of fixing.
His interesting code uses the wave event spos value to cleverly adjust the left
starting point of the wave event so that it does not need to start 'before' the part,
thus no left-side 'hidden events' gray arrow need be drawn.
However, this only works up to a point. You can only shrink a wave event so much
approaching zero length. If the left part border is moved to the right so much that
a wave event can't be shrunk any more, there's no more choice but to leave
the event's length alone and declare its position to be 'before' the left part border,
which should turn on the left-side 'hidden events' gray arrow.
This is exactly what I have done. i re-enabled the left-side 'hidden events' gray arrow,
which had been disabled.

In order to accommodate event positions that are 'before' the left part border,
we talked about this, we need signed position values. Our current unsigned values
simply are not going to cut it here - there's no negative range.

I mentioned recently that I had created typedefs in muse_time.h
typedef int64_t MuseFrame_t;
typedef int64_t MuseCount_t;
typedef int64_t MuseTick_t;
in the hopes of someday replacing ALL unsigned or signed int positional variables
and usages with these typedefs.
But with thousands of places using signed or unsigned ints for positional calculations,
that's a gargantuan task. Although, I have begun to use them in certain places.

The idea was to expand the number of bits to accommodate the new required negative sign.
I had been very reluctant to simply change our 32-bit unsigned integers to 32-bit signed.
That's because doing so effectively halves our maximum recording time.
It sacrifices an entire range of numbers just so we can have a few rare negative
'before the part' event times !
However, saying that MusE's current maximum recording time is a 32-bit unsigned value
is quite dubious and only theoretical at best, because there are already many places
in the code where the position values are treated as signed values, such as for the purpose
of differential calculations.
I was worried that half the recording time would mean only several hours instead of
tens of hours. Ardour likes to say that you can record an entire festival with that time.
But my worries are misplaced - let's be honest, MusE is not really for recording
an entire festival, it is for recording much shorter individual songs.
In fact ideally the software would detect when the end is approaching and
restart the transport with a new recording track, like a GoPro camera's FAT32 4GB
file limit where it automatically breaks up the recordings.
Also, using internal 64-bit positional values may not jive neatly with code that uses
libraries or controls that use 32-bit values.
You'll notice I've been very careful to respect our current unsigned range.
I've been very reluctant to upset that wide range. I avoided treating the values as signed.
But this made code more complex, requiring two sections of code for every
differential comparison or operation of these unsigned values.

So... For the above reasons, I hereby heartily endorse treating all our unsigned integer
positional values as signed integers.

Go ahead, cast them as signed immediately when working with them.

This has made it much easier to proceed with improving @spamatica 's
left border resizing code. Simply treat all positions as signed values.

When we are working in there, let's try to remember to ensure that
no recomputed position value goes beyond signed int maximum and 'wraps around'.
We must ensure for example that no new event or part is allowed to be placed
at beyond signed int end, or drawn on a canvas.
In other words, we must enforce that upper signed int limit as being the true
upper end of our time range.

It means we must now state for certain that MusE's position range is no longer
the theoretical 32-bit unsigned range, but in fact truly only 32-bit signed range.

It's too bad that this really only needs to apply to events inside parts.
Parts that start 'before' a track would not make much sense - but - actually I
can see a possible need for that down the road with latency correction and so on.
And, as with events, we already treat part positions as signed in some places.
So we might as well be consistent across the board in our new declaration of positional range.

Concepts sound OK so far?
Cheers.
Tim.

[spamatica]

So... For the above reasons, *I hereby heartily endorse treating all our unsigned integer positional values as signed integers*. Go ahead, cast them as signed immediately when working with them. This has made it much easier to proceed with improving @spamatica <https://github.com/spamatica> 's left border resizing code. Simply treat all positions as signed values. When we are working in there, let's try to remember to ensure that no recomputed position value goes beyond signed int maximum and 'wraps around'. We must ensure for example that no new event or part is allowed to be placed at beyond signed int end, or drawn on a canvas. In other words, we must enforce that upper signed int limit as being the true upper end of our time range. It means we must now state for certain that MusE's position range is no longer the *theoretical* 32-bit unsigned range, but in fact truly only 32-bit signed range. It's too bad that this really only needs to apply to events inside parts. Parts that start 'before' a track would not make much sense - but - actually I can see a possible need for that down the road with latency correction and so on. And, as with events, we already treat part positions as signed in some places. So we might as well be consistent across the board in our new declaration of positional range. Concepts sound OK so far?

Sounds workable to me. As for the new limitation I concur it is mostly theoretical. If my calculations are right it will still give more than 6 hours at 96khz (and personally I think 96khz is overkill, but anyway). Which for MusEs purposes really should be considered enough. And sorry for the bug cleanup, I have a tendency for cutting corners, I know that...

[terminator356]

I think the situation was left unclarified when we talked about it.
I mentioned the unsigned values and questioned how we were going to handle these special events.
They would have appeared waaay at the very end of the song.

Note that I am talking about our Pos class values (parts, events, calculations etc),
not transport positions.

Our transport position system remains unchanged for now.
The negative position concept might be extended for use in our transport system.
Allowing for example to queue something to begin playing before transport time zero,
to overcome playback latency. i.e. The transport and even our timelines could have an adjustable offset.
But one big problem I forgot to mention is that Jack for example hands us
32-bit unsigned frame values. What would we do when we are handed a transport
re-positioning command beyond our 'new' 32-bit signed limits?
Ignore it? Maybe. Wrap it around by making it negative? No.
Limit it to half of an unsigned range? Can't. Transport wants to go to that place. Other clients
may go there.

How can we go to a place we cannot accommodate with 32-bit signed values?
(That's one reason I chose large 64-bit signed integers in muse_time.h typedefs that I mentioned).

There are other places that might need or want to accommodate this new negative position concept.
Such as the BBT/FRAMES positional control widgets.
They could show that an event has negative position.
Although - we can't actually show the hidden event anywhere so how can we click on it?
So we rely on those 'hidden events' arrows to inform the user to expand the part.
We could have parts auto-expand-with-enforced-limits and click-to-expand etc.

It's really a big scary task, I hate having to do this, but I hope these initial steps are
a reasonable solution to the immediate problem for now.

All for now.
Tim.

[terminator356]

Go ahead, cast them as signed immediately when working with them.

Actually, it might be better to remember to cast them as my muse_time.h typedefs,
such as MuseFrame_t. That way, if we ever do get around to changing the fundamental
Pos value types, these working casts will be ready for them - the casts would become benign.

Ugh. Well, back to my commits...

[terminator356]

Ehm, well. All that stuff sounded great but...
I ran into trouble much sooner than I was expecting.
While working on that border resizing stuff and negative event times with respect to the left border,
I suddenly realized that we cannot just cast our unsigned integers as signed (32 or 64 bit) and work
with them as easily as that, even though some of it appears to be working correctly here.
Of major concern is our class EventList and other maps that have unsigned integer for key.
EventList's map key, methods, arguments, and members and everything associated with them
must be changed to signed integer.
Which means everything else in the program must change. Class Pos and everything associated
with it. Functions. Helpers. Comparisons. Timings. And so on.

Here's another thing: Even with all those changes, if we do wish to use 64 bit signed integers,
which I highly recommend to cover the whole range, then simple casting from unsigned int to int64_t
may not be enough in some cases. I think there are two ways to cast our unsigned integers:

(int64)(x) -> Fills upper bits with zero. Simple cast. Great for preserving all unsigned ints in a loading song.
(int64)(int32)(x) -> Converts to signed 32 and then to signed 64. Might want this for loading songs
which might already contain these 'negative unsigned integers' we have been using recently.

When working with say, libraries or sound servers that use 32-bit unsigned positions, casting
may also be of concern if we use signed (32 or 64 bit) numbers.

I've not given up. Sorry for the delays. I really am sorting all of this out. Realizing a lot of things.
Three different iterations of the code so far. Much thinking time... Great you guys are keeping up.

There is no time right now to do massive changes to structure.
So for now I will limit all events times to zero. No negative event times. No events before a part's left border.
Sucks, I know. We would love to be able to shape a part into a 'window' onto some events,
especially clone parts - that's where such a feature would shine and make sense.
But I'll make up for it by providing two (and only two) modes:

1. Drag events with border.
2. Do not drag events with border. (Like we have now). Left border hidden events not allowed,
   all events will be moved forward if necessary.

FYI: Also, there are picky concepts with clone parts, which I think Robert realized in his coding:
When resizing a part's left border, if not dragging events with the border, all clones must be done.
When resizing a part's right border, if dragging events with the border, all clones must be done.
Otherwise you can choose (currently via ctrl) to operate only on the one clone.
It's real goofy and awkward, to explain to the user why ctrl only works in some cases.
The reason for all of this is parts lack an events offset variable.
Thus clones parts' events times must all be the same (referenced to the left border).
Thus we cannot allow clones to have independent event times.
Thus we cannot fully realize our desire for clone parts to be 'windows onto events'. Right border only.
If parts had an events offset variable, we could do this.
Not to mention, all that other signed integer stuff above would be required for negative event times.

See ya.

[spamatica]

Hi Tim, great to see you still chugging along. Took a walk earlier, good winter weather, -15C, sunny and no wind. I recommend!

Not sure I understand all the implications (especially on part with clones). But I do understand the impossibility of fixing everything but instead at least taking steps in the right direction.
I'm reminded of the midi recording modes you had worked a bit on, overdub etc, is it a part of this larger work? In any case I suppose it will be yet some time before you have anything to push.

[terminator356]

No, this is strictly border resizing stuff.
Overdub is in another branch here, waiting to be worked on again... Hopefully soon.