defacto2 / df2 Goto Github PK

Add datalist value to the format flag for both df2 html group and df2 html people.

This can be used for list forms that input both groups and people.

<datalist id="newFile-group_brand_for-list">
  <option value="$YN (Syndicate BBS)" label="$YN (Syndicate BBS)">
  <option value="0AC" label="0AC"><option value="0x0007" label="0x0007">
</datalist>

The algorithm is currently broken due to some past modification.

Currently the formatting of Ordinal Numbers are broken.

• 4Th Dimension
• 5Th Dynasty

Multiple groups separated by , commas; the second and following groups are not capitalised for key words such as 'the'.

• Fairlight, the Silents
• Lightforce, Pact, Trsi, Venom
• Razor 1911, the System
• Rebels, 2000Ad

Abbreviations are not capitalised, such as 4Ad instead of 4AD; 2000Ad instead of 2000AD.

• 2000 Ad (12)
• 2000Ad

Create a command to run through and list (with a URL and file count) all the groups that share very similar names.

• trailing letter swaps, such as Coolz and Cools
• single letter addition, such as Cool and Cools
• near identical sentences, Lucid Dream BBS, Lucid Dreams BBS
• multiple words joined together, such as 2000 AD and 2000AD
• hyphenation, 2000-AD vs 2000 AD vs 2000AD
• phonetic swaps, f vs ph
• similar fonts, i vs l vs 1, 5 vs 7, 0 vs o
• numbers vs digits, 2 vs two, 7th vs seventh

The sitemap should also include all static URLs created by the site and they should listed be at the start of the file. The can fetch and use the last modified date value of the /view/file.cfm.

Also include wayback links and get the lastmod of the index.html, index.htm or whatever is given in the database URL.

df2 output groups --cronjob

Embed the following HTML/CFML into the output.

Organisation.cfc

var query = model("Organisation").getGroups(key=params.key,returnAs="query",includeSoftDeletes=includeSoftDeletes);
variables.listOfResults = groupsListed(query)
listOfResults = listSort(listOfResults, "text", "asc")
if(!ListLen(listOfResults)) return render404();

organisation/list.cfml

<!--- pagination statistics --->
<div class="pagination-statistics">
	<span class="label label-default"><!---#listLen(listOfResults)# --->#button.badge#</span>
	<cfswitch expression="#params.key#">
		<cfcase value="bbs">
			<span class="label label-warning mobile-hide" title="'The Example BBS' will be listed as 'Example BBS'">'<i><b>The</b></i> '' prefix has been dropped from board names</span>
		</cfcase>
		<cfcase value="ftp">
			<span class="label label-warning mobile-hide" title="'The Example FTP' will be listed as 'Example FTP'">'<i><b>The</b></i> '' prefix has been dropped from site names</span>
		</cfcase>
		<cfdefaultcase></cfdefaultcase>
	</cfswitch>
</div>

The command should scan through every entry flagged as a compressed file or archive.

Easy to follow guide:
https://www.honeybadger.io/blog/golang-logging/

Panic in golang is similar to the exception. Panic is meant to exit from a program in abnormal conditions. Panic can occur in a program in two ways

Runtime error in the program
By calling the panic function explicitly. This can be called by the programmer when the program cannot continue and it has to exit

https://golangbyexample.com/panic-and-recover-golang

df2 approve

❯ df2 approve
───────────────────────────────────
262 new files approved
9 database new records were skipped
───────────────────────────────────
        all good, there is nothing to do `files.filename`
        all good, there is nothing to do `files.group_brand_for`
        all good, there is nothing to do `files.group_brand_by`
        group fixes, all good, there is nothing to do
        initialism fixes, all good, there is nothing to do
no people fixes needed, time taken 0.7 seconds

df2 new

❯ df2 new
INFO    SCAN for new general submissions and cleanup
        all good, there is nothing to do
        1. 40060  19 Dec 2021  ID:303417  skipped, no changes needed ✓
        TOTAL, 1 Demozoo items
        time taken 0.5 seconds
INFO    SCAN for new proof submissions and cleanup
        TOTAL, 4119 proof records
        all good, there is nothing to do
INFO    SCAN file archives
        1. 43893  13 Apr 2023  Warez.PC.Game.Starforce.Keygen.Collection-TWC.7z
ERROR   zipcontent record error, archive read: read uuid/filename: readr: system reader: system could not read the file archive
        2. 43894  13 Apr 2023  Warez.PC.Game.CD.Keygen.Collection.20211008-TWC.7z
ERROR   zipcontent record error, archive read: read uuid/filename: readr: system reader: system could not read the file archive
        all good, there is nothing to do
        TOTAL 0 archives
        time taken 0.2 seconds
INFO    GENERATE any missing screenshots or thumbnails
        1. (2320) Frx-Cls1.jpg 59 kB ✗
        2. (2321) if-class.jpg 92 kB ✗
        3. (2322) if-cls2.jpg 56 kB ✗
        4. (2323) p-cls01.gif 25 kB ✗
        5. (2324) p-cls1.gif 18 kB ✗
        6. (2325) p-cls.gif 22 kB ✗
        7. (2326) p-hit.gif 33 kB ✗
        5267. (43573) BACK.BMP 197 kB ✗
        5268. (43640) opg.jpg 40 kB ✗
        5269. (43654) DP-RISC.RIP 6.0 kB ✗
        5270. (43770) cdmhqbbs.jpg 144 kB ✗
        5271. (43771) cdm-nodes.jpg 28 kB ✗
        5272. (43772) cdm3.gif 11 kB ✗
        5273. (43773) cdm2.gif 9.1 kB ✗
        5274. (44795) 2802.png 16 kB ✗
        5275. (44796) 2764.png 8.4 kB ✗
INFO    GENERATE any previews for the text files

        32. (45028) steinberg_wavelab_v5_01a-zone.zip 14 kB
ERROR   readme file cannot be blank or invalid
        1654. (42896) vor-0794.zip 226 kB
        2543. (40441) notices.zip 1.4 MB
        4902. (37295) WEHRMCHT.zip 1.7 kB
        8599. (22264) A.NOTE.ON.A.RUSSIAN.PUBLIC.WEB.GROUP.TEAM.TSRh.AND.LEADER-progopis.rar 5.3 MB
        10884. (14248) Beware.of.GioTiN.aka.y0tek.aka.ptrJ.READ.NFO-iND.rar 20 kB
        11421. (12472) UNPs.zip 18 kB
ERROR   readme file cannot be blank or invalid

        11631. (12077) DBC_Tuts.zip 73 kB
ERROR   readme file cannot be blank or invalid

        13942. (8899) TCPAK2.ZIP 2.9 kB
ERROR   readme file cannot be blank or invalid

        18235. (65) NWR-netmonkey.weekly.report.zip 5.6 MB
ERROR   readme file cannot be blank or invalid
        SCAN 10 fix attempts for 18288 text files
INFO    FIX Demozoo conflicts
        all good, there is nothing to do
INFO    FIX malformed database records
        all good, there is nothing to do `files.filename`
        all good, there is nothing to do `files.group_brand_for`
        all good, there is nothing to do `files.group_brand_by`
INFO    FIX malformed database groups
        group fixes, all good, there is nothing to do
        initialism fixes, all good, there is nothing to do

df2 proofs are designed to handle large numbers of files that spam the console.

Instead of listing every file just update the single line with the current file.

Prefix that update string with a counter listing success, skipped and failed proofs.
Make sure to get the width of the console and always clear the whole line whenever updating.

• First list the active item number, the total number and filename.
• Then dynamically change the activity to only display one at a time, file_zip_content > jpeg > webp > 400 > 150 > txt etc.
• List a green tick after each updated activity or a red cross if it fails.
• Also prefix a spinner and time taken?

[SPINNER? 10 minutes] 0010 ✓ 0000 X | 0001/0953 (30039) blahlbah-byblah.zip ✓ file_zip_content

Have option to display id/uuid/filenames that failed and at what point?

Complete the --pouet flag so it can be used with a cronjob.

scan the demozoo api for missing or deleted pouet ids

• approve command should support music files that have required tags.
• fix database command should apply intro tag to all music platform.

Create an automated synchronisation with Demozoo.

For optimisation collect the all existing DZ ids into a searchable array.

Specifically looking to sync the following productions.

• Windows

• • cracktros
    https://demozoo.org/productions/?platform=1&production_type=13

• MS-DOS

• • cracktros
    https://demozoo.org/productions/?platform=4&production_type=13
• • bbs
    https://demozoo.org/productions/?platform=4&production_type=41

Production List

http://demozoo.org/api/v1/productions/?page=1 >> http://demozoo.org/api/v1/productions/?page=2712

{
  "count": 271197,
  "next": "http://demozoo.org/api/v1/productions/?page=2",
  "previous": null,
  "results": [
    {
      "id": 00001,
      "title": "blahblah",
      "platforms": [
        {
          "url": "http://demozoo.org/api/v1/platforms/4/",
          "id": 4,
          "name": "MS-Dos"
        }
      ],
      "types": [
        {
          "url": "http://demozoo.org/api/v1/production_types/13/",
          "id": 13,
          "name": "Cracktro"
        }
      ]
    }
  ]
}

• df2 fix approve > df2 approve

When handling proofs record 30444 c94b4e71-9d43-4ab2-90bd-a61e46ca5884 returned ...

unarr: No valid RAR, ZIP, 7Z or TAR archive

• Have the application generate a JSON with filename and date that will be cached by nginx.
• Let the user browser parse the JSON to render the view.
• Have the images be exported to WebP, have a PNG fallback and resized while kept in a special directory, /images/newest_16/.
• Compress the JSON.
• Include all required information in JSON including last updated information.

Examples:

go run -race . o d --cronjob

Use a simple ping or get to test all API endpoints. The results could be displayed with the -version flag.

Usage a wrapper to the more reliable and expansive imagemagick cli tool when parsing or generating images fail using Go image libraries.

https://github.com/gographics/imagick

https://github.com/quirkey/magick (last update: 2014)

Create a tool to scan through a group hosted on DZ and add the productions to Defacto2.

For example: https://demozoo.org/groups/85899

https://demozoo.org/api/v1/releasers/85899/
https://demozoo.org/api/v1/releasers/85899/productions/

New features to add.

MOVE: /opt/backup/sql move out of date SQL files to ~/downloads to free up space.

MOVE: /opt/daily-defacto2/ROOT/incoming/user_submissions to a TAR archive in ~/downloads .. append oldest and newest created dates to the filename.

example: incoming-user_submissions-08sep2019_10mar2020.tar

df2 demozoo --new
Requires the command to be rerun when new submissions are first encountered. A notification with a counter should be displayed after the Total Demozoo items handled: xx string.

Commands that require webp conversion require the download of 3rd party libraries. At the moment these are stored in /vendor in the active directory. This means multiple vendor directories are left on the file system based on where ever df2 is run.

Instead .df2.yaml should supply a vendor directory and by default it should be located in ~/.df2/vendor.

example:
df2 fix text -s false

Implement screenshot support for the following formats and extensions.

• PDF
• BMP
• RIP (BBS vector support)

Using SQL Boiler.

This should be quicker than using the web interface.

Any changes should also update the updateAt.

df2 config info

connection: up ✓

to

connection: alive ✓

fix demozoo data conflicts
moved 0 Demozoo #releaseadvert records to #groupapplication
moved 0 Demozoo #releaseadvert records to #releaseinstall
fix malformed database entries
no fixes needed, time taken 0.0 seconds

Move this to df2 approve.

txtwebp: to webp mimetype ".gif" != .png .jpg .jpeg .tif .tiff .webp: unsupported image format

Example file: uni-1293.zip

Example id: 42869

Create a command to ping the various URLs on the site to confirm they return a 200 status with an expected title value.

URLs to ping

• all site areas and tables that can be fetched
• various URL containing unusual chars. for users and people
• include site wide static URLs
• a few light downloads of item files, text files and images
• test 10 random file urls using an array of ids.
• test a couple of disabled file records to confirm 404.

Ping both localhost and remote domain URLs to confirm the issues are not DNS or cache issues.

On Linux, with the default terminal, the progress indicator uses an unsupported font.

df2 output groups -c

querying  (··ᗧ••••••••••••••••••) 9.44 %

Source:
pubfor "MotorSoft" pubby "M&N BBS"

Process:

INFO > FIX malformed database groups
✓ "M&N BBS" ⟫ M & N BBS (1)

Result:
pubfor "M & N BBS" pubby "M & N BBS"

df2 clean --delete

clean sum calculate: sum calculate: scan path backup: backup part: remove "downloads/fff9de63-ff96-4aab-8738-c4275f509c3f.txt": remove downloads/fff9de63-ff96-4aab-8738-c4275f509c3f.txt: no such file or directory

Returns exit 0 ?

Apply a recover to this routine,

mm3crk.zip4 items, text file: MM3.NFOgoroutine 1 [running]:
/home/runner/work/df2/df2/main.go:59 +0x45
/home/runner/work/df2/df2/vendor/github.com/mholt/archiver/zip.go:409
github.com/mholt/archiver.(Zip).Extract(*

runtime/debug.Stack()
        /opt/hostedtoolcache/go/1.20.3/x64/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
        /opt/hostedtoolcache/go/1.20.3/x64/src/runtime/debug/stack.go:16 +0x19
main.main.func2()
        /home/runner/work/df2/df2/main.go:59 +0x45
panic({0xd31a40, 0x163e3b0})
        /opt/hostedtoolcache/go/1.20.3/x64/src/runtime/panic.go:884 +0x213
github.com/mholt/archiver.(*Zip).Walk(0xc0004e0de0, {0xc0002de190?, 0x10054c0?}, 0xc00039e608)
        /home/runner/work/df2/df2/vendor/github.com/mholt/archiver/zip.go:409 +0x15d
github.com/mholt/archiver.(*Zip).Extract(0xc0004e0de0, {0xc0002de190, 0x43}, {0xc0004e238d?, 0x0?}, {0xc0005585a0, 0x18})
        /home/runner/work/df2/df2/vendor/github.com/mholt/archiver/zip.go:450 +0xcc
github.com/Defacto2/df2/pkg/archive.Extractor({0xc000256826?, 0x6?}, {0xc0002de190, 0x43}, {0xc0004e238d, 0x7}, {0xc0005585a0, 0x18})
        /home/runner/work/df2/df2/pkg/archive/archiver.go:38 +0x1d2
github.com/Defacto2/df2/pkg/zipcontent/internal/record.(*Record).Textfile(0xc00039ed40, {0x1005ea0?, 0xc00009c008?}, 0xc00039ec80)
        /home/runner/work/df2/df2/pkg/zipcontent/internal/record/record.go:158 +0x25a
github.com/Defacto2/df2/pkg/zipcontent/internal/record.(*Record).Archive(0xc00039ed40, 0xc00007a340, {0x1005ea0?, 0xc00009c008?}, 0xc00039ec80)
        /home/runner/work/df2/df2/pkg/zipcontent/internal/record/record.go:119 +0x274
github.com/Defacto2/df2/pkg/zipcontent/internal/record.(*Record).Iterate(0xc000256820?, 0xc00007a340, {0x1005ea0?, 0xc00009c008?}, 0xc00039ec80)
        /home/runner/work/df2/df2/pkg/zipcontent/internal/record/record.go:89 +0x359
github.com/Defacto2/df2/pkg/zipcontent.Fix(_, {_, _}, _, {0x1, 0x0, {0xe0f288, 0xd}, {0xc00002606b, 0x3}, ...}, ...)
        /home/runner/work/df2/df2/pkg/zipcontent/zipcontent.go:74 +0x7e9
github.com/Defacto2/df2/cmd/internal/run.genZIPList(...)
        /home/runner/work/df2/df2/cmd/internal/run/run.go:363
github.com/Defacto2/df2/cmd/internal/run.New(_, {_, _}, _, {0x1, 0x0, {0xe0f288, 0xd}, {0xc00002606b, 0x3}, ...})
        /home/runner/work/df2/df2/cmd/internal/run/run.go:405 +0x37d
github.com/Defacto2/df2/cmd.glob..func17(0x164d640?, {0xdd4bc3?, 0x0?, 0x0?})
        /home/runner/work/df2/df2/cmd/main.go:53 +0x185
github.com/spf13/cobra.(*Command).execute(0x164d640, {0xc00008c200, 0x0, 0x0})
        /home/runner/work/df2/df2/vendor/github.com/spf13/cobra/command.go:944 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0x164d640)
        /home/runner/work/df2/df2/vendor/github.com/spf13/cobra/command.go:1068 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
        /home/runner/work/df2/df2/vendor/github.com/spf13/cobra/command.go:992
github.com/Defacto2/df2/cmd.Execute(_, {0x1, 0x0, {0xe0f288, 0xd}, {0xc00002606b, 0x3}, {0xc00002a09b, 0x8}, {0xdede6f, ...}, ...})
        /home/runner/work/df2/df2/cmd/main.go:71 +0xdc
main.main()
        /home/runner/work/df2/df2/main.go:100 +0xa98

A go library to render progress bars in terminal applications
https://github.com/gosuri/uiprogress

A really basic thread-safe progress bar for Golang applications
https://github.com/schollz/progressbar

multi progress bar for Go cli applications
https://github.com/vbauerster/mpb

Go (golang) package with 70+ configurable terminal spinner/progress indicators.
https://github.com/briandowns/spinner

Terminal spinner package for Golang
https://github.com/tj/go-spin

A very simple spinner for cli Go apps
https://github.com/caarlos0/spin

A simple, configurable, multi-platform spinner
https://github.com/leaanthony/spinner

A minimalistic emoji package for Go (golang)
https://github.com/enescakir/emoji

Create a command to dump the SQL database for use in the Docker images.

docker_assets/import-data/dump.sql

To keep things simple, use the mysqldump app to generate the dump as a dependency.

Maybe, also compress the file using a text friendly archiving tool?

Specifically the ones installed for the terminal on the Ubuntu OS. These should be a last resort when all other Go specific methods fail.

All archive tools should be attempted regardless of the file extension.

Add support for:

• UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP. /usr/bin/unzip
• UnRAR UNRAR 5.61 beta 1 freeware Copyright (c) 1993-2018 Alexander Roshal /usr/bin/unrar

There's an issue when scanning the DZ API. The scan is discovering existing entries on DF2 that are missing DZ IDs but is not checking for existing Pouet IDs on the API result.

So two identical then entries exist in the DF2 database.

• The original Df2 entry with an empty DZ record and a pouet ID.
• And a new Df2 entry created from the DZ API result that includes both Pouet and DZ ids.

The HTML template is incorrect. It should follow this layout.

<div class="pagination-statistics"><span class="label label-default">434 ftp sites</span></div>
<div class="columns-list" id="organisationDrillDown">
<h2><a href="/g/2nd-front-ftp">2nd Front FTP</a> <small>[2NDF]</small> <small>(1)</small></h2>
<hr>
<h2><a href="/g/a_one-ftp">A-One FTP</a> <small>(1)</small></h2>
<h2><a href="/g/absolute-zero-ftp">Absolute Zero FTP</a> <small>(1)</small></h2>
...
</div>

-t, --target string what file section to clean
options: (default "all")

This should list all options.

df2 clean

Generate type-safe Go from SQL

https://github.com/kyleconroy/sqlc

Complete all the Flags for the df2 output people command.

They should match df2 output groups.

There should be outputs for 4 files.

artists, coders, musicians, writers