During Manual Testing I discovered adding my files to folklike shows a error in the form that the filetype is not valid see attached

Create a System Model and Database Entry. This allows setting system variables for a variety of tasks. Such as read-only.

• Read Only
• System Module
• System Module Tests

This module is a reusable bit of code that can be used in any part of the rails application. These validations go along with validations that are inside of the various parts of the form builder.

• Phone Number : validates that a phone number is a legit phone.

• Limit Words. Limits the number of words in a string.

• Limit Characters. Limits the number of characters in a string.

• URL. Validates the string is a proper URL. Accepts telnet, ssh, fpt, https, http.

• Optional URL. Validates the string is a proper URL or is an empty string.

• Email. Validates that the given param is a valid email. Uses base ruby regex for doing this.

• WVU Email Validates the internal email address.

• IP Address. This one had a to-do to fix because it wasn't working properly so this should be fixed using ruby-core classes.

• Integer. This was not also fully functional in MFCS. A valid integer could have float point numbers and other constants and the MFCS regular expression didn't take this into account.

• Integer with spaces. This has been fixed to allow for many different test cases to pass and uses a shared function with the other method.

• Alpha Numerical Validation. Validates a variety of alphanumerical characters including spaces.

• Alpha Numerical Validation with spaces. Validates a variety of alphanumerical characters including spaces.

• Alphabetical Characters Only.

• Alphabetical Characters Only with spaces.

• No Special Characters.

• Date Validates as a edtf date. Edtf will silently fai as nil if the edtf date is not valid. See specs for full examples of valid edtf dates.

Code Can be found https://github.com/wvulibraries/mfcs-ruby/blob/master/app/lib/validator.rb

Tests and Specs
https://github.com/wvulibraries/mfcs-ruby/blob/master/spec/lib/validator_spec.rb

Validation Duplication Check

When checking the database for duplicate entries, the database validation is not able to determine if this is the same record that has already been created, so it creates an error saying that the item may not be saved. This renders all item updates completely useless as no items will update.

JavaScript

This will be the same in the JavaScript because the JS does not know when the item is being updated or saved or any information about the item.

• create a link to a form
• create a link to field names
• Modify API to use Headers or Params
• Adjust HTML to create Target for form and field
• Create Event Binding for Form and Fields
• create a way to gather manual choices
• create a way to save manual choices
• display and edit manual choices
• check that from and field are present so that item creates a valid link

Bugs Fixed :

• Saving the form the loading data doesn't trigger the change event.
• Changing to each object Manual Items are Dropped
• Default Button When Reprocessed.

Form Builder

JavaScript Bugs

• Fire Change Event at Pageload for: Title, Display Title, Description, Submit Button, Update Button
• When Metadata Form is selected in the Form Builder remove the option for WYSIWYG, FILE UPLOADS, and SELECT OPTIONS.
• When Digital Object Form is selected in the Form Builder Add the option for WYSIWYG, FILE UPLOADS, and SELECT OPTIONS.

Validation

• An IDNO Validation needs to take place to be sure there are not more than one IDNO fields attached to the form.
• Check field names for duplicate field names within the form.

Interface Simplification

• During the building events we need to lock the IDNO name field so that it doesn't conflict.
• When modifying a metadata field, IDNO should not show up.
• When an IDNO is added the IDNO field should disappear as an option to use.

Form Validation Bugs that had to be tested on the server and in the interface.

• Metadata Field
  • If is a metadata field must have one field
  • Duplicate Field names are not allowed
  • No file fields or other extraneous fields are allowed
• Digital Object Field
  • Idno must be present
  • Duplicate names are not allowed
  • Must have at least an idno and a title

Modeling

A project model needs to be made that can be used to attach forms, users, and objects independently.

• User: When a user selects projects all data entry associated with that user is labeled to that project by default.
• Form: When a project is attached to the form that project tag is included in all objects made under that form.
• Objects: When an object belongs to a project those objects can all be exported to that project tag.

Users

• USER A I select that I'm working with Project B
• Every Item i create needs to be tagged Project B.

Form

• Then every object in that form needs tagged to that project.

Object

• The object itself can have added projects associated with them directly after creating them, but on the update process.

Interfaces

There needs to be a project tab at the top of the page for each user that pulls their specific projects and pushes their specific projects. AJAX.

Admin:

• CREATE
• READ
• UPDATE
• DELETE
• API

View:

• Project Objects
• Project Users
• Project Forms

The validations need to be considered in each form section before the item can be saved. Not sure how I'm going to work this in currently.

FORM BUILDER

• [name] - can not have spaces
• [choices] - must have fields selected also
• [file types] - must be a valid file type

The form builder group has some custom JS Validations that need to be added to be sure that these items are present.

• Url
• Trailing Spaces
• Starting Spaces
• Regular Expression

DATA ENTRY

FIELD SPECIFIC VALIDATIONS

• Detailed Error for the validation issue at hand.
• Textarea / Text Fields (Min, Max) for each format (words, and characters)
  • Between Min / Max for Characters
  • Between Min / Max for Words
• Number (Min, Max, Step)
  • Number Step is divisible by the entered value
  • Number is in between Min / Max
• Any Field (no duplicates modifier)
  • Checks Database for Duplicate Values

VALIDATIONS

• Date (EDTF)
• Email
• Website
• Character Limit
• Word Limit
• Alphabetical Characters Only
• AlphaNumerical
• AlphaNumerical Plus Spaces
• Numerical
• Numerical Plus Spaces

REFACTOR

• Namespaces these items into classes which create an instance that will allow us to perform the needed tasks.
• Actor This is the performer part of the namespace and it uses Single Responsibility Principle to call the other namespaced methods from the information provided by the form field.
• Validator This class checks the hash and returns the items as an array that need to be validated
• TextLength
• Integer Between Range
• Integer Step Length
• Regular Expression
• Date (EDTF)
• Email
• Website
• Character Limit
• Word Limit
• Alphabetical Characters Only
• AlphaNumerical
• AlphaNumerical Plus Spaces
• Numerical
• Numerical Plus Spaces

CURRENT MFCS BEHAVIOR

• The current behavior only shows the validations on submission and doesn't describe the error.
• The error is provided on the top of the screen providing the name of the field only.
• Text validations only work for min/max words and the validation in the validation dropdown menu.
• Integers do not validate min, max, or step. Behavior is somewhat deprecated because it is never used (maybe because it never worked?).

NEW BEHAVIOR STEPS

• Create JavaScript Event Controller for Validations
• Use Timeout and Debounce to keep from submitting many times on keyup
• Create Controller and API for Validations
• Test Controller for API Validations
• Use JavaScript to return the error and show the validations.

THIS IS FOR FORMBUILDER ONLY OPTIONS
THESE HAVE BEEN COMPLETED. NOT BEEN DONE IN THE BACKEND JUST THE JS.

• Audio
  • Bitrate Changes
  • Format Changes
  • Web Quality Preview
  • Audio File Thumbnail
• Video
  • Wide Screen
    • 240p: 426x240 (16:9)
    • 480p: 854x480 (16:9)
    • 720p: 1280x720 (16:9)
    • 1080p: 1920x1080 (16:9)
  • Standard Definition
    • 426x320 (4:3)
    • 640x480 (4:3)
    • 854x640 (4:3)
    • 1280x960 (4:3)
  • BitRates
    • 700-12000 kbs
  • Video Screen Capture for Thumbnail
    • Size, DPI, Time, Type of File

• Metadata Forms
  • Get Name, ID, and DisplayName
• Fields
  • Linked Form to a Field Name
  • Dealing with Nested API Items

Form Builder

-- MOVED TO FORM BUILDER BUG

Object IDNO Number

• Case 1 User sets IDNO, number needs to be checked against all current IDNO's from items in that form.
• Case 2 System sets IDNO based on a pattern (ex: ###### from wvcp) (ex: iai######) (ex:###), each number symbol should be incremented based on the last item that exists in the database and should be implemented right before the save.

Logic

User Supplied IDNO

• Sets IDNO Directly using form input

System Supplied IDNO

• Sets IDNO based on Form Pattern Supplied
• Checks Last IDNO in system, increments, and sets that to the pattern

Both

• Validation Check to be sure it doesn't conflict with any other IDNO in the form.

Idno::SystemIdno Model

• Initializes and sets up the pattern and incremental versions
• Creates and IDNO number based on the count and padding supplied.
• Identifies the Padding by indexing the pattern
• Creates a fully formatted IDNO to match the pattern
• Count System Records for that

Item

• Set IDNO before validation
• Set IDNO in external hash if it is the proper form.

The database migrations from PHP are going to be handled by Tracy... I will be working on the import scripts for ruby which will require cross walking data to new data models.

Items

Metadata

Controller

• Items::Metadata

ROUTES

• "/items/metadata"
  • This view should be a list of metadata forms that the user can create new items with.
• "/items/metadata/:form_id"
  • This view shows all the items that are associated with that form_id. Might have to look into pagination for this.
• "/items/metadata/dups(/:form_id)"
  • This view shows the duplicate objects for the form.
• "/items/metadata/new"
  • Redirect back to the main page and request the proper form Id.
• "/items/metadata/new/:form_id"
  • Opens a new item for the form page using the form builder to build the form.
• "/items/metadata/:id/edit"
  • Opens a specific Item in the form for editing.
• "/items/metadata/:id"

Digital Objects

Controller

• Items::DigitalObjects

Routes / Views

• "/items/digital_objects"
  • This view should be a list of digital object forms that the user can create new items with.
• "/items/digital_objects/:form_id"
  • This view shows all the items that are associated with that form_id. Might have to look into pagination for this.
• "/items/digital_objects/dups(/:form_id)"
  • This view shows the duplicate objects for the form.
• "/items/digital_objects/new"
  • Redirect back to the main page and request the proper form Id.
• "/items/digital_objects/new/:form_id"
  • Opens a new item for the form page using the form builder to build the form.
• "/items/digital_objects/:id/edit"
  • Opens a specific Item in the form for editing.
• "/items/digital_objects/:id"
  • This should view the digital object and list all revisions.
• "/items/digital_objects/revisions/:id"
  • Should open and show a specific revision.

The FFMPEG Video class needs the ability to make thumbnails, set size, format, and keep aspect ratio among other video options set in the MFCS form.

Shared

• Base Shared
• sets to file, from a file, and options for the command
• Set validations to make sure the output, file, and options.
• establish where FFmpeg is on the system.
• the filename minus the extension.

Metadata

• Get File contents metadata for stream items such as resolution.|

Thumbnail

• Thumbnail Creation
• Disabled Audio
• Number of Frames
• Set the size of the thumbnail keeping the aspect ratio of the original file dimensions.
• Set the time for the frame to grab.
• Give it a perform command so the command runs.

Video

• Size
• Bitrate
• Optional way to add new commands.

When an object is uploaded the following Image Tasks May need to take place.

• Create Thumbnail
• Web Preview
• Export conversion
• OCR
• Combine PDF
• Single PDF (Mulitple FIle Uploads)

Automatic Importing and Adjusting of MFCS data that was exported from MFCS old to use in the new MFCS.

This entails that imported data works perfectly and as expected in the new form builder so that if edited the item is modified correctly.

The allowed file types is a dynamic multiple fields that should store as an array in the JSON of the file field.

• Dynamic Toggle Allowed File Types
• Edit / Update

Form Builder Navigations need to be remade so users can create and modify a list of links to show beside the form. These links will be custom form. They need a way to be reordered and modified to link to a variety of forms. The person who modifies this will need to be responsible for making a list of evolving tasks.

Because drag and drop is not accessible(directly) may want to look at another way to sort the fields. However, GitHub does use it so maybe accessible methods can be added or multiple ways of doing it to keep from causing issues.

• Drag and Drop Event Listener Research

  • created UX/UI Wireframes
  • functional mapping made
  • Accessible choice options for non drag and drop users.

• Programming the Drag and Drop

  • created a stimulus code pento work on trying events faster
  • learned how to fire the following events
    • drag, dragstart, dragend drop, dragleave, dragover
  • working on the proper data store of the dragged element
  • figure out how to move the element within the dom and delete the original element
  • establish a workable pattern for sorting the items, maybe using an array and array index.
  • create matching touch events for all drag events that way mobile users can use the same functions
  • establish a way for the moving or active item to become keyboard movable
  • Accessible choice options for non drag and drop users.

Bugs Fixed:

• CSS Padding Bug
• Pass array of elements to the global scope to use across multiple events
• Fix the lagging of the drag and drop by switching the elements over from the drop event to the drag over event.
• Create a representation sort order that also adjust the form if any field is currently active.

The form was not rebuilding on a new/error or updated error of the form. We had to recall a method as json, instead of getting the field from the active record and parsing it to json.

System / Job Management

MFCS has a ton of scripts that maybe with MFCS or on the MFCS server that include backup, reprocessing, and other tasks.

Evaluate Scripts
Account for All Jobs
Rewrite Jobs

Form Builder is full of features and probably a little bloated, but we are trying to match exact functionality minus anything that is not actually being used.

Form Builder Features

• ALL Fields
  • Template for HTML
  • Bindings for Inner JS
  • Bindings for Complete JS and JSON Storage of Information
  • Preview Binding
  • Field Edit Binding
  • Delete
  • Help Preview
• Choices Fields
  • Manual
  • Form Link
  • Multi Select
  • Async Ajax Functions for Form Field Linking through API
• Help Field
  • 3 Different Help Types
  • 3 Different Help Messages Type / Entry Forms
  • HTML Santization
• Metadata Schema Links
  • Enumerable List of Metadata Types
  • Corresponding Qualifier and TItle
  • Ability to add more than one mapping to every single form field.
• IDNO
  • Manual
  • System
  • Must be a type that can be verified as unique to the database.
• Text Fields
  • Character Count Limit (Validation
• Validation Option
  • Regex
  • List of Enumerable Validations
  • Valid Regex
• Sorting
  • Drag and Drop
  • Accessible Option
  • Resorting if gapped Numbers

The audio class needs to be similar to the video class but slightly modified.

Shared

• Base Shared
• sets to file, from a file, and options for the command
• Set validations to make sure the output, file, and options.
• establish where FFmpeg is on the system.
• the filename minus the extension.

Audio

• Bitrate
• Optional way to add new commands.

Hey, I'm Camillo from CodeClimate, I'm reaching out about the ZenDesk ticket created here, and I would be glad to help with your issue.

I took a look at your failed job in CircleCI and I realized that you may be missing a setting in your yml.

The trick here is to set CC_TEST_REPORTER_ID at the moment SimpleCov is being run, meaning setting for both bundle exec rake and the CodeClimate action. (in your setting file you had it only in the latter)

As you can see in the simplecov ReadMe JSON formatter section, there is a comment explaining that SimpleCov will automatically use the JSON formatter along with the HTML formatter when the CC_TEST_REPORTER_ID variable is present in the environment.
Newer versions need that JSON formatter to work. You may find more context about this in this comment.

Please let me know if this works for you 😄

Currently, we have a TDD test suite for Rails and Ruby Modules, but nothing set up for behavioral tests and JavaScript related tests. Need to setup a capybara instance and a virtual chrome for doing JavaScript-based interaction tests. This will be a little bit of a learning curve for drag/drop, click, and other customized data inputs. It may also be a time-consuming process because of the number of possibilities for each form and field.

This is a brief summary of the frontend component work that has been done up to this point.

The process of creating MFCS has come down many roads and those choices mainly affect the frontend. The first of those being that we are using the new rails pipeline for delivering assets, which is essentially using a heavily modified webpack. Webpack is a very specific way of interacting with the JavaScript, CSS, and some HTML. In doing so we have our views which will have references to our frontend modules and hopefully reduce complexity and overhead in the app in future aspects.

Modules Completed

• Alerts (CSS, JavaScript and HTML)

• Breadcrumbs (CSS, JS, and HTML/ERB)

• Buttons (CSS)

• Events (JS)

• Footer (CSS, JS, and HTML/ERB)

• Form Builder (CSS, JS, and HTML/ERB)

• Forms (CSS)

• Header (CSS, JS, and HTML/ERB)

• Home (CSS, JS, and HTML/ERB)

• Login (CSS, JS, and HTML/ERB)

• Main (CSS, JS, and HTML/ERB)

• Top Bar (CSS, JS, and HTML/ERB)

• Validation (CSS / JS)

• Variable (CSS, JS, and HTML/ERB)

• WVU Branding (CSS, JS, and HTML/ERB)

Responsibility for this class is to assume responsibility of checking file type and calling the proper conversion class after the type of the file has been checked.

• Create Types
• Create a way to determine Type
• Create Tests
• Use established builder pattern.

Basic Work

• Models
  • Schema
  • Tests
  • Additional Logic
• Views
  • Create / Edit
  • Show To Admins
• API for use in Form Builder

Form Builder

• Form HTML
• AJAX to API
• Preview of Image

Conversions

Process

1. Form is made with options for file conversions.
2. New Object is added to the form.
3. File Upload happens and Media Object is made.
4. Job is added to sidekick for processing image conversions.
5. Job calls the image conversion actor class which determines which conversion class needs to be fired based on the media objects mime type.
6. Form Info is passed in as options to extrapolate what conversion needs to be done.

Image Conversions

• tif, tiff, jpeg, jpg, gif, pdf, png

Audio Conversions

• mp3, wav, wma

Video Conversions

• wmv, mov, mp4

Conversion Process Diagrams

API Version 1 is a mock-up of some API functionality that was needed for form building along with a few functions that were in the MFCS repo in a variety of places.

Case 1: In the includes folder there existed a series of JSON output files that were created by MFCS for use within the form builder. These essentially were PHP output files with no protection except for a script that allowed no view access off-campus.

Solution:
We needed a way to use this and be sure that only we can use this part of the application. I created a public API and a private API module and worked to get the information out safely and securely.

• Public API Module

• Private API Module

• Controller for API

• Routes and Namespacing

• Queries and Documentation

• Testing

The form builder is being worked on as an operation form element, in MFCS this is done in multiple PHP VIews, this logic needed to be abstracted to presenters and logical models.

In the new MFCS these have been abstracted out to Modules that fit somewhere between a Presenter and Actor.

• Base Presenter
• Base Field Model
• TextField
• TextareaField
• NumberField
• DropdownField

--- More complicated Ones that need JS Hooks.

• MultiSelect Field
• File Upload Field
• Wysiwyg Field

Media

This is the model we are going to use to associate the files to items. The idea is to have this data in a few locations just in case because the previous system if this information was lost from the item it was not recoverable.

Previous System

• saves the archival path
• does not save a relation back to the item
• saves json in the item json for the file path and information

New System

• saves the file to a archival path and working path, only using the working path for the modifications to the file.
• save based on the UUID as in the previous system, but keep this information all located in an active record model
• save the item id in the media model so that there is another way to find it.
• create a returnable method for the paths and create a to json method that gathers all the information for the items submission.

Database

• id :bigint
• checksum :string
• file_errors :jsonb
• filename :string
• mime_type :string
• path :string
• size :string
• uuid :string
• virus_scanned :boolean
• created_at :datetimenotnull
• updated_at :datetimenotnull
• form_id :integer
• item_id :integer

Methods

• to_json
• uuid_path
• conversion_path
• archival_path
• working_path
• tmp_path
• export_path

The file upload box needs to contain a variety of checks and balances for each possible upload. An example could be configurations for images, audio, video, etc. Each one will be different as per the type of upload box that is in the form.

• Images
  • Rendering / Transformation
  • Allowed File Types
  • Watermarks
  • Default File Convesions
  • Thumbnails
  • PDF
    • OCR
    • Mege to Single PDF or Multiple PDFS
• Audio
  • Bitrate
  • Clips
  • Conversion Types
  • Allowed File Types
• Video
  • Bitrate
  • Clips
  • Conversion Types
  • Allowed File Types
  • Thumbnail Capture

This is mostly done, clips is a suggestion that isn't currently in MFCS, going to skip that for now since we have so much more work to do on this project.

We need to figure out a way to create a testing suite for the JavaScript side of things. Some of this should be covered by behavioral testing, but some need to be functional because of return values.

An active record model concern that can attach to all models for a modular way of checking if the system is in readonly mode before doing any actions that involve databases.

• Creates a user model for allowed users who are able to use the system.
  • Database or OmniAuth ( :cas )
  • Validation of firstname, lastname, and username
  • Enumerations that allow the user to be a regular user, an editor, or an admin.
  • Interface that allows for either login type.