Readme for Tur-Space.

What is it?
It is a script to either delete or archive releases from your incoming
sections.

But there are 100 other scripts that does the same thing?
True. But neither of them did what I want. Or, if they did
they were too advanced for me to tackle.
I just wanted my own that I could mold as I liked.

So whats so special?
Well, this one works like this.
You define one or more incoming drives. Like this:
TRIGGER=/dev/hda1:25000:30000

That means it will check /dev/hda1 - Once it gets 25000 MB (25GB) free, it will be processed.
It will delete or move releases until there are 30000 MB (30GB) free.

As you can see, it works pretty much like Zios Spacemaker. Why? Because I love that one but
it has one or two fault in my case. 
Lets say I have several DIVX archive disks thats not raided.
Say I have DIVX1 -> DIVX5 archive. Zios spacemaker would move from the incoming section to
DIVX1. If there is no space in DIVX1, it will move the oldest from there to DIVX2. 
If there is no space in DIVX2.. etc

Tur-Space moves to the defined archive with the most free space.
If none of the defined archives have enough space it will
find the oldest release in all of them and delete it.
If theres still not enough space, it will delete the oldest release in the 
dir it just deleted the last one in, until there IS enough space.

So, instead of Incoming -> DIVX1 -> DIVX2 -> DIVX3.
its Incoming -> DIVX1. Incoming -> DIVX2. Incoming -> DIVX3.

Personally, I think thats better. Atleast in my case.

Also, Tur-Space use 'tuls'. A stand-alone ls binary so it should work on all distros.

Ok, some basic stuff. How to set it up.

You should have some knowledge of the df binary and grep. Atleast its recomended.

#------------------------------------------------------------------------------------------------#

Start with editing tur-space.sh. The only setting here is where the config is. Previosly,
tur-space.sh tried to locate the file if it was in the same dir as tur-space.conf, but that did
not work too well everywhere, so now you have to hardcode its location.

The config:

[IGNOREDIRS]
IGNORE="GROUPS lost\+found"

In IGNORE, you set dirs that should never be moved. In this case, GROUPS and lost+found.
It is case sensitive and matches whole dir. Example: New-GROUP-News would still be deleted/moved.
Seperate dirs with a space.


[VARIOUS]
TULS=/glftpd/bin/tuls

This script requires tuls. A stand-alone ls binary made by cruxis.
It can be found on my site at http://www.grandis.nu/glftpd
Make sure it works first and then specify its location here.


TMP=/tmp

A dir to store temporary stuff in.


LOGFILE=/glftpd/ftp-data/logs/tur-space.log

File to log to. This is NOT to glftpd.log


GLLOG=/glftpd/ftp-data/logs/glftpd.log

If you want announce to irc on move and delete, point this to your glftpd.log
file and read README.announce


COPY_OPTIONS="-rf --preserve=timestamps"

We do a cp on the releases (copy) and the verify that its all there.
Once verified, it runs rm -rf on the releases in its original location.
This is where you set which options the cp binary runs with.
The default '--preserve=timestamps' only seems to work on RedHat and
perhaps some other distro. The only setting that MUST be here is -r for
recursive copy. Oh, and unless you want to anwer "yes" to questions, -f is good too =)
For Debian, you apparently want to use "-rf --preserve" - Thanks DaShizNit.


CHOWN_OPTIONS="0:400"

If you want it to change the owner of the release after its moved, set
the options for chown here. The default '0:400' means in MY case
glftpd:SiTEOPS. 400 might not be the GID for your siteops groups.
You can check the GID for groups in /glftpd/etc/groups.
You can not use the name here, unless you know what you're doing.
Put a # infront of this one to disable running chown on the release.
This is actually whats added as 'chown $CHOWN_OPTIONS dir', so you can
add -R or whatever your system uses for a recursive chown, if you like.


CHMOD_OPTIONS="755"

If you like to change permissions on the release after its moved, set the
chmod options here. Should be a straight forward option.
Put a # infront of this one to disable changing permissions on it.
This is actually whats added as 'chmod $CHMOD_OPTIONS dir', so you can
add -R or whatever your system uses for a recursive chmod, if you like.


DATE_BINARY=""

If you're using FBSD, you need to download and compile sh-utils and specifically
'gdate'. Its a GNU compliant date binary.
You then specify the path to gdate here. tur-space requires your date binary
to support '-d' as in:
"-d, --date=STRING         display time described by STRING, not `now'"
Leaving this empty will just use 'date' from your path. Should be fine for most
linuxes.

DU_BINARY=""

Same as DATE_BINARY.. Used for FBSD because it needs a gnu du binary that supports
du -csm (default du for FBSD dosnt support the m flag for MB output).
Leave empty if you're running linux. Can always check if your current du binary
supports it by running:
du -csm /tmp | tail -n1 | cut -f1
It should then return the size if the /tmp directory in MB.


MAX_LOOP="10"

There is a safety check in tur-space. It can only delete 10 items in your archive
to make room for any one release. Ie, it will loop "until there is enough space to
hold the release". Here you can modify that value incase you feel the need to.
Incase you specify the wrong device / path for an archive, it will remove stuff from
one directory, then check space again on another device. Since this havent changed, it
will continue to delete stuff. So this is only for safety.


SECURITY_PATH="/glftpd/site/.*/"

This is a security. Nothing outside of this can be deleted and its just a safety
precation. Use .* to specify anything, so /glftpd/site/.*/ means inside any subdir
of /glftpd/site only.


[TRIGGER]
TRIGGER=/dev/hda1:25000:30000

Incoming sections are defined here. One TRIGGER per disk.
You may add more then one TRIGGER. Example:
TRIGGER=/dev/hda1:25000:30000
TRIGGER=/dev/hdb1:15000:20000

So
TRIGGER=device:Free_Space_in_MB_To_Start_Moving_or_Deleting_On:Amount_of_Space_in_MB_Required_Before_It_Quits

It must start with TRIGGER=
The '/dev/hda1' is the device to check space on. It must appear when
running 'df' and only on one line. Run: 
df | grep "/dev/hda1"
and make sure it only returns one row.

It can be any part of the df output, as long as its unique for that mount.
Example: Say you have this line in your df output.
192.168.0.15:/shares/5  196015808  45292840 140765888  25% /glftpd/site
Thats a NFS share.

Something unique would be /shares/5
So:
TRIGGER=/shares/5:25000:30000

Do NOT define archive disks/mounts here. They will be freed of space, if needed, automatically.
Note that they are seperated with a :, so the device part can not contain a ':'.


[INCOMING]
INCDVDR=/dev/hda1:/glftpd/site/DVDR:
INCDIVX=/dev/hda1:/glftpd/site/DIVX:
INCXBOX=/dev/hda1:/glftpd/site/XBOX:
INC0DAYS=/dev/hdb1:/glftpd/site/0DAYS:DATED

Ok, here you specify the incoming sections on the TRIGGER devices.
They must start with INC, followed by the name of the section
The name can be anything but it must be unique for each section.

So, its INC<name>=Device:Full_Path:

The Device must match the device specified in one of the TRIGGER above.

As you can see from the INC0DAYS example above, DATED is added after the
last :
With DATED, it will find the oldest release by sorting by number instead of
date. The first number in the dated dir must be the month in 01-12 format, ie
0101 or 01-January etc, for DATED to work.

If DATED is not defined (just a : at the end), it will find the oldest release
by sorting on date.

Dont forget the trailing :
Dont specify a trailing / on the path ( /glftpd/site/XBOX/ = baaad ).

The name here must be unique. You can NOT have two INCDVDR=
If you do have two incoming drives for DVDR, add them using
INCDVDR1 and INCDVDR2, then when it comes to the archive, make two setups. One
for DVDR1 and one for DVDR2, pointing to the same destination dirs.


[ARCHIVE]
ARCDVDR=/dev/hdf1:/glftpd/site/Archive/DVDR/DVDR1
ARCDVDR=/dev/hdg1:/glftpd/site/Archive/DVDR/DVDR2

ARCDIVX=/dev/hdg1:/glftpd/site/Archive/DIVX/DIVX1
ARCDIVX=/dev/hdh1:/glftpd/site/Archive/DIVX/DIVX2
ARCDIVX=/dev/hdi1:/glftpd/site/Archive/DIVX/DIVX3

ARCUTILS=/dev/hdj1:/glftpd/site/Archive/ISO-UTILS/ISO-UTILS1
ARCUTILS=/dev/hdk1:/glftpd/site/Archive/ISO-UTILS/ISO-UTILS2

ARC0DAYS=/dev/hdn1:/glftpd/site/Archive/0DAYS/0DAYS1/2004

These are the archive sections for the INC sections above.
Note that they must start with ARC, followed by the same name
you used in the INComing sections. That is how it matches the INComing section with 
the ARChive section.

As you can see, you can specify more then one ARC<name> section
for each INC<name> section. 

The device the disk/mount to check if theres space for it and after the : is 
the actual path it will move the releases to.

As with the INC<name> sections, the device can be anything unique in the df
output, that does not include a ':'.

******************************************************************************
In fact, if its mounted as above, the path itself should be unique. If so, you
can even use this format:
ARCDIVX=/glftpd/site/Archive/DIVX/DIVX1:/glftpd/site/Archive/DIVX/DIVX1
ARCDIVX=/glftpd/site/Archive/DIVX/DIVX2:/glftpd/site/Archive/DIVX/DIVX2
ARCDIVX=/glftpd/site/Archive/DIVX/DIVX3:/glftpd/site/Archive/DIVX/DIVX3
******************************************************************************

It will then try and find the archive with the most space free in it when moving.
If no archive have enough space to move the release to, it will find the oldest 
release in all of the archives.

Once that is found, it will be deleted.
If there is still not enough space, it will keep deleting from the archive
it deleted the oldest one from, until there is enough space, or until it has deleted
the MAX_LOOP= amount of releases.

As you see, there is no ARCXBOX for the INCXBOX defined. That means
releases from the INCXBOX section will simply be deleted.

Thats all there is to it ! 

#------------------------------------------------------------------------------------------------#

Now, run it without arguments first. You'll see you have 2(3) options.

Start with ./tur-space.sh sanity
It will show the configuration and find the oldest releases in each INC section.
It will also show you any apparent errors, like if you have ARCTV but no INCTV, etc
( since that would be useless ).
Make sure to check the top of the sanity check too for the chown/chmod options set. 


Next, run ./tur-space.sh go debug
*******************************************************************************
That will NOT move anything. But it will act as if it does and show you exactly
how it works.

Since its not actually moving anything, space used etc will not change but it will
still act as if the release it said it moved was (wont try to move something twice etc).
It will break automatically after 20 loops so just let it work and then have a look 
at the result.
It will also log to the LOGFILE as if it ran live.
*******************************************************************************

If it says that there is already enough space on the TRIGGER device(s), you can change how
much space it should trigger on and run debug again. Just change it back before running it live.

Once you are satisfied, just run ./tur-space.sh go

You can crontab it to run 'go' as often as you like. If there already is enough
space on all the TRIGGER devices, it will just quit.

#------------------------------------------------------------------------------------------------#

NOTES:
It has a lockfile, so you can not start it twice. However, should a bad error occur, like a
configuration error, it will quit without removing the lockfile.
If you run it and it just dies, check with debug and it will show you the last error in 
the LOGFILE. Of course, you can check the LOGFILE yourself too.

If one of your ARC<name> sections can not be found (unmounted perhaps) and it needs to delete
something to fit a release, it will NOT do it, even if you have other ARC<name> that is also
an archive to the same section.
It will still move stuff to the other ARC<name> sections if there is enough free space though.
This is to prevent it from deleting stuff because a mount failed or HD died etc.

This is also displayed when running with 'sanity'. Try to add an ARChive with a non existing 
device and you'll see what I mean.

To disable a TRIGGER, INC<name> or ARC<name>, just put a # infront of it.
Experts: It does a: for each in `grep "^INC" $config`, so #INC is not detected.

Feel free to add any comments you like to the config. Just remember that
comments may NOT start with: IGNORE, TULS, TMP, LOGFILE, TRIGGER, INC, ARC, COPY_OPTIONS,
CHOWN_OPTIONS & CHMOD_OPTIONS (etc) or they will be considered an argument for tur-space.sh to read.

It actually does not move releases right away. It copies them, using the $COPY_OPTIONS.

After it has copied the full dir, it will check that the dir it copied has the same amount of files in it
as the dir in the incoming section.
If it does, it will rm -rf the dir in the incoming section and consider it done.

However, if the copied dir does NOT contain the same amount of files, it will wait 5 seconds and then
check again (to allow any cache to clear) and if its still not the same, it will log it and quit.
This is one of those exits where it leaves the lockfile to prevent any damage. You'll have to check
why it did this manually and fix it yourself. 

#------------------------------------------------------------------------------------------------#
One last thing. This is a script I made for me. It works how I want it to, so I wont modify it 
because you want it to work in some other way. Unless you find any bugs, dont bother me about it. 

You also can NOT hold me responsible if you configure it to delete your entire box, or if a bug
does it. Sorry, but it can happen I guess.
#------------------------------------------------------------------------------------------------#

/Turranius - 2004/2005