A template driver script to update and maintain a daily archive of files
This README serves two purposes: to describe the template daily-driver script and how to use it, and to serve as a sample README for your own projects that you create using daily-driver as a template. The daily-driver template provides a script that is useful for running software that updates or maintains archives of data organized by date. If you want to write code that takes a date argument and then creates or updates a set of files, and you want to run this code automatically (e.g., as a cron job) with self-healing capabilities to keep track of gaps in the archive to try and fill in later runs, than this template is for you.
- Bash
- Add other software you use in your project
This section provides information for how to get daily-driver set up and working on your system. Be sure to add additional content based on the project you build from this template!
This application is designed to run in the bash shell. The bash shell is available on Linux and MacOS systems, on Windows systems via Windows Subsystem for Linux, and on ChromeOS via the Crostini (Debian Linux based) container.
The daily-driver application relies on other software that needs to be installed on your system.
See Git - Installing git for more information about installing git on your system.
- Clone the repo:
git clone [email protected]:avram-meir/daily-driver.git
- Nothing else needs to be done to get daily-driver working, but add any additional installation instructions here for your own project.
This section provides information about how to use the daily-driver application. Update and modify this for your own software project.
The entire functionality of daily-driver is governed by a bash script: daily-driver.sh
. This script can be run on cron to update your date-based archives with new data, once you modify its child script update-archives.sh
to do what you want it to do.
Features of daily-driver.sh
:
- Run with no arguments, will run
update-archives.sh
passing it today's date via a-d
option (e.g.,update-archives.sh -d "`date +%Y%m%d`"
). - To run different dates or a range of dates, supply a
-d
option. - To have
daily-driver.sh
check your archives for a set of expected files and only runupdate-archives.sh
for dates when those files are not there, supply a-c
option. - To have
daily-driver.sh
check and update the archives (or just runupdate-archives.sh
) for a number of days prior to the earliest date in its list, supply a-b
option. - To keep track of dates where archive files fail to update or
update-archives.sh
returns a non-zero status (failure), supply a-l
option.
As an example of how to use this script, consider this crontab entry:
0 0 * * * /path/to/daily-driver/daily-driver.sh -c /path/to/daily-driver/archive.config -b "-30 days" -l /path/to/daily-driver/missing_dates.txt
This usage would, daily at midnight, check the archives for missing files for the 30 days prior to today's date, today's date, and any dates in /path/to/daily-driver/missing_dates.txt
, running update-archives.sh
for any dates where missing files are found. The result of update-archives.sh
would be checked as well as the archive to make sure the files got created, and any dates with problems or missing files would be written back to /path/to/daily-driver/missing_dates.txt
for the next day's run.
Usage
./daily-driver.sh [options]
Options
These are the options available for daily-driver.sh
.
Option | Description |
---|---|
-b <string> | Pass a valid GNU date delta (e.g., '-30 days' or '10 days ago') and the script will scan the archive described by the configuration file provided through the -c option for missing files for that many days before the earliest date supplied through the -d option. If no -c option is supplied, the script will run update-archives.sh for each of those days. If no -d option is supplied, the date delta will apply to today's date (e.g., ./daily-driver.sh -b '-30 days' will run update-archives.sh for the 30 days leading up to today's date and today as well). |
-c <filename> | Pass a configuration file that lists all of the expected files in the archive. When updating for a date, the script will check for the existence of these files. If they do not exist, then the script will run update-archives.sh to try and create them. If they do exist, the script will do nothing and move to the next date in the list. See below for the format of the configuration file. |
-d <date> or -d <date1 date2> | Pass a date to update, or a start date and stop date to update for a range of days. The dates should be provided in a format understood by GNU date's --date option. |
-h | Print a usage statement and exit. |
-l <filename> | Pass a file containing a list of dates to update. Any dates where update-archives.pl fails will be written back to this file, while dates that have a successful run (or already have existing files in the archive defined by the -c option) will be removed from this file. This is a good way to keep track of gaps in your archives. |
Archive configuration file (what you pass with -c
)
The file passed using this option must provide a bash variable $files
that contains a list of the archive files you want daily-driver.sh
to check for. As this file is executed by daily-driver.sh
, any bash script formatting can be used to set up this list. Each file in the list is passed to the GNU date command via [+FORMAT], so date FORMAT wildcards can be used in the filename. So, for example, if your configuration file contained the following:
archive_path='/path/to/archive'
files=${archive_path}/%Y/%m/%d/file1.txt
files+=${archive_path}/%Y/%m/%d/file2.txt
And you ran ./daily-driver.sh -c /your/config/file -d 20220501
, then the script would check for the following files:
/path/to/archive/2022/05/01/file1.txt
/path/to/archive/2022/05/01/file2.txt
And would run update-archives.sh
to try and create these files if they were not there.
Missing date list file (what you pass with -l
)
The file passed using this option must provide a bash variable $missingdates
that contains a list of the dates that you want update-archives.sh
to check and run. This file need not exist, and if dates where archive updating fails occur during run time, they will get written to this file, overwriting whatever was there before. Note that this file is executed as a bash script by daily-driver.sh
, so any bash script formatting works.
An example file created by daily-driver.sh
when it tried and failed to update an archive for the first five days of May, 2022:
missingdates=(20220501 20220502 20220503 20220504 20220505)
The driver script daily-driver.sh
calls a second script, update-archives.sh
and passes it a date argument (-d <date>
). In this template, the script does nothing functional, but this is where you add your own functionality when building a project from this template. If you pass a -c
argument to daily-driver.sh
, update-archives.sh
should be written to create or update those archive files.
Here is where you'll likely be adding your project's awesomeness to the README.
See the open issues for a list of proposed features and reported problems.
Any contributions you make are greatly appreciated.
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature
) - Commit your Changes (
git commit -m 'Add some AmazingFeature'
) - Push to the Branch (
git push origin feature/AmazingFeature
) - Open a Pull Request
Adam Allgood