[TangerineSDR] [HamSCI] Bash Scripting Help

[Rob Wiesler]

On Sat, Sep 10, 2022 at 11:50:04 -0400, Robert McGwier wrote:
> On Tue, Sep 6, 2022 at 12:12 PM Jonathan <emuman100 at gmail.com> wrote:
>> I need some assistance in writing a bash script that will run as a cron
>> job to archive VLF data. The data is recorded in hour-long chunks as files.
>> I have a cleardown script that will cleardown the oldest files and always
>> maintain a rotating buffer, so as the newest file is being recorded, the
>> oldest file is being deleted. What I need now is a way to locate the oldest
>> file, encode it to flac using vlfrx-tools vtflac utility, then delete the
>> original file. Here is an example:
>>
>> find locating the oldest files. There is often more than one and the
>> naming convention is YYMMDD-HHMMSS. Running the following:
>>
>> find /data/vlf_32k -type f -mtime +50
>>
>> prints:
>>
>> /data/vlf_32k/220904-144416
>> /data/vlf_32k/220904-150000
>> ...
>> /data/vlf_32k/220904-230000
>>
>> Next, we need to encode those files to flac to the archive directory:
>>
>> vtflac -e /data/vlf_32k/220904-144416 >
>> /mnt/archive/vlf_32k_flac/220904-144416.fx
>> vtflac -e /data/vlf_32k/220904-150000 >
>> /mnt/archive/vlf_32k_flac/220904-150000.fx
>> ...
>> vtflac -e /data/vlf_32k/220904-230000 >
>> /mnt/archive/vlf_32k_flac/220904-230000.fx
>>
>> Finally, delete the original files:
>>
>> rm /data/vlf_32k/220904-144416
>> rm /data/vlf_32k/220904-150000
>> ...
>> rm /data/vlf_32k/220904-230000
>>
>> I haven't done a lot of bash scripting, so I'm asking if anyone here may
>> have experience with this. I need the above commands to be executed by a
>> bash script so it'll complete those commands no matter how many files that
>> the find utility will find. Sometimes, there will be no files the find
>> utility locates and the script should just exit. It'll be executed every
>> hour by cron. Any help or assistance will be appreciated!
>>
>> Thank you!
>>
>> Jonathan
>> KC3EEY
>>
>> --
>> Please follow the HamSCI Community Participation Guidelines at
>> http://hamsci.org/hamsci-community-participation-guidelines.
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "HamSCI" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to hamsci+unsubscribe at googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/hamsci/CAOY0kB3J3HX_8fFgNBgxzmCE2C3aVvPQO%3DNa54TiZX8RK23U-Q%40mail.gmail.com
>> <https://groups.google.com/d/msgid/hamsci/CAOY0kB3J3HX_8fFgNBgxzmCE2C3aVvPQO%3DNa54TiZX8RK23U-Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>
> #!/bin/bash
> dirs=($(find /tmp/test -type d))
> for dir in "${dirs[@]}"; do
>   cd "$dir"
>   ls -pt | grep -v / | tail -n +4 | xargs rm -f
> done
> replace /tmp/test with your directory name.

Okay, no, this will break.

First, $dirs will break if any directory in /tmp/test has whitespace in
its name.  This is one of the reasons I didn't answer with a standalone
script - bash and POSIX shell scripts have difficulty iterating over
filenames safely.  This *particular* issue comes from your find(1)
invocation, but once you fix that you'll find yourself trying to make
sure the read builtin works properly with null delimiters, and it's just
cleaner not to do that.

Second, don't just blindly cd.  cd has options, and since your dirs var
may contain fractional directory names due to the aforementioned bug,
you may end up in the home directory just before that rm -f .  A
directory named "/tmp/test/ -L" will cause exactly that to happen.  Use
the -- convention to force all subsequent arguments to be treated as
filenames.  Normally this wouldn't be a problem because find(1) will
ensure that all the files it prints will start with a known prefix, but
you should get into the habit anyway.

Third, *any* use of ls ... | [ tail ... | ] xargs ... is a bug for lots
of reasons.  If you really want to delete a bunch of files in a
directory, use find(1) again.  That means something like:

find -maxdepth 1 -type f -exec rm -f {} +

... and I continue to assert that deleting input files in batches is
wrong.  Delete them incrementally when you finish processing them (which
is skipped in this script anyway).

Fourth, you could have replaced *all* of this script with this
"one"-liner, which is both safe and clean (and at least as correct as
before):

#!/bin/sh
find /tmp/test -type f -exec rm -f {} +

Fifth, it's not clear to me at all that it's correct to recurse into
subdirectories in the first place.


The most important thing, though, is whether or not Jonathan has gotten
something to work, which he has so far declined to mention.

--
73,
Rob Wiesler (AC8YV)