Coder Social home page Coder Social logo

phetsims / paper-land Goto Github PK

View Code? Open in Web Editor NEW
12.0 2.0 2.0 147.55 MB

Build and explore multimodal web interactives with pieces of paper!

Home Page: https://phetsims.github.io/paper-land/

License: MIT License

Shell 0.02% JavaScript 99.85% CSS 0.09% HTML 0.04% Procfile 0.01%
codesign community javascript multimodal open-source paper ar augmented

paper-land's Introduction

Paper Playground

Documentation Matrix Server Youtube Channel License

logo

Interactive Play Meets Multimodal Web Experience

Paper Playground is an open-source project for collaboratively creating multimodal web experiences by means of mapping JavaScript* code to real pieces of paper and manipulating the code in your physical space.

(*written directly or using a no/low-code component-style interface!)

We aim to support a community interested in bringing physical interaction as a means to collaboratively solve problems in codesigning virtual technology.

Paper Playground is based on the Paper Programs open-source project and has been extended to incorporate the library stack used by PhET Interactive Simulations as a convenient 2D scene creator and manager. The project focuses on enabling quick prototyping of web projects, with a particular emphasis on easy addition of audio features (like sounds and sonifications), speech description (both TTS engines and screen reader-like descriptions), and other non-visual features that are often difficult to design and develop alongside visual elements in these projects.

The ideas behind Paper Playground are simple, but the possibilities are infinite.

  1. Create programs, as many as you would like, to represent your project.
  2. Map data variables, parameters, and functions to paper position, size, shape, rotation, proximity, and use optional marker positions for more functionality.
  3. Print out the dot-covered papers that belong to those programs.
  4. Put those paper programs in front of a webcam.
  5. Watch and interact with the output of that code in your browser.
  6. Move programs around and trigger the mappings between your papers position in space and your code!
  7. Change your program code, make new papers, and keep the creativity going!

Paper Playground setup detecting paper programs that create a moon lander and control vertical thrust

Installation

See the Paper Playground Docs website for the most up to date installation instructions.

Paper Playground setup detecting paper programs that create a moon lander and control vertical position

What is coming?

Our team is focused on a few large initiatives for integrating other projects into Paper Playground and for making it more friendly for non-technical, non-JavaScript users and designers.

⭐ If these projects interest you, please join our community and take part in the development and discussion!

  1. πŸ”“ Creator Interface: Abstracting the JavaScript code through means of a no/low-code component style interface that assembles the data components of your programs and highlights the relationships between your programs (e.g., At a quick glance, what information is needed and passed between programs?). Check out our progress by opening the Creator page in the tool!
  2. πŸ€– AI Support: Using LLMs to support a user answer the question: "How do I turn my idea for a multimodal interactive into data variables, functions, and entire programs?". We are working toward this goal by adding an optional AI assistant inside the Creator page to help generate custom JavaScript code for more complex functions!
  3. πŸ“€ Easier API integration for paper programs to control or output to other browser displays (besides our scenery-focused Display and the legacy canvas-focused Projector), as well as new inputs such as microcontroller integration via Bluetooth and WebSockets. See our example with the Weavly educational coding project!

However, there are many other areas that the project can be expanded (see Community below for Contributing guidelines and suggestions)!

🌏🌎🌍 Join our Community! 🌏🌎🌍

πŸ“‹ Roadmap

Docs

πŸ“š Updated setup and tutorial!

Tool

πŸ“ƒ More examples in the hosted database highlighting the power of multimodal design!

πŸ’» A UI for creating basic Papers without deep JavaScript knowledge!

πŸ€– Investigating use of LLMs to help users go from ideas to your suite of paper programs!

License

This software is licensed under the MIT license. For more information, see the LICENSE file.

Acknowledgments

This tool was adapted from Paper Programs by JP Posma, which was inspired by the work at Dynamicland.

paper-land's People

Contributors

brettfiedler avatar dependabot[bot] avatar janpaul123 avatar jbphet avatar jemgold avatar jessegreenberg avatar jonathanolson avatar osnr avatar paulsonnentag avatar sgwilym avatar shaunlebron avatar stevekrouse avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

paper-land's Issues

Can global methods for managing config be removed?

At the end of client/camera/entry.js` are the following definitions for functions that would be globally available within the camera window:

window.getPaperProgramsConfig = () => JSON.parse(localStorage.paperProgramsConfig);
window.setPaperProgramsConfig = config =>
  (localStorage.paperProgramsConfig = JSON.stringify(config));

However, I think these may be vestigial, since it looks like there are methods for storing the config that are passed to CameraMain, and the current config is available in the state to CameraMain. I'm also not finding any usages of them. Can these be removed to avoid any confusion?

How can we support local, custom asset upload and usage?

We discussed the idea of allowing users to add custom asset files (images, sounds, etc) for use in their programs. Currently, those must be added to the file directory of paper playground itself.

We discussed:

  • Including some preloaded assets from our examples (e.g., parachute person, loon, some phet sounds).
  • Giving an interface on e.g., the editor page to open a file directory and upload:
    (opening for instance, this dialog
    image
    )
    1. Server storage of uploaded assets (I believe MIT Scratch does this. Something about an sb3 file was mentioned) $$$
    2. Local storage, perhaps browser storage that would be unavailable if they changed browsers and lost if they cleared their browser cache.

This question also hinges a bit on if we later wish to keep this tool as a tool run from the command line or if their is a "built version". If the former, we could just let them upload things to an asset directory and have git ignore it?

Duplicate Programs in one Playspace from another

From #8:

We should have one playspace that contains all programs, one "vetted" space with templates, and then infinite others.

We'll need a way for spaces to reach across each other, e.g., when in summer-codesign-2023, we should be able to pull scripts from approved-templates.

It'd be nice to have a good way to see programs in other spaces without necessarily leaving the one you're in.

The do-no-further-development "simple" implementation here I believe, is:

  1. Create new Program in Playspace 2.
  2. Navigate to Playspace 1.
  3. Filter/Search for Program you want from Playspace 1.
  4. Copy entire Program
  5. Navigate to Playspace 2
  6. Paste entire program
  7. Modify as desired

If Programs are backed up to Github, then you could also copy-paste from the hardcoded examples in the repo.

Maybe this is enough, but there are a few "luxury" features we could implement:

  • [simple or already in place?] Editor opens on last opened Program when switching Playspaces
  • Create another interface component that lets you preview another Playspace and duplicate a Program into current Playspace
  • Enable side-by-side editor (which I believe Monaco Editor supports, at least in the newer versions) where the two editors can exist in two different playspaces.

Change the name of the repository

We would like to open up the repo going down the line. Papyrus is a fun name, but it doesn't align with the branding of the tool. It also overlaps with the Papyrus coding language and several design tools, including at least one UI design tool.

@jbphet indicated it is not too much work to change the name, but we should do it sooner rather than later.

Just a few names to toss into the ether:

  • paper-playground (This has stuck around the most so far and may be the default choice)
  • codevise (also has code in the name) + suffix (PG, Playground, JS)
  • interaction-playground
  • copaper
  • collabopaper
  • paper-n-play
  • codesign-playground
  • script-collab
  • collabosense + suffix
  • collaboraction + something
  • playper

Originally posted by @BLFiedler in #26 (comment)

Ping everyone to discuss. If no preference, we'll go forward with the default: paper-playground.

When settled, I'll assign JB/JG to make the change.

Lots of unused things in AudioGraphSoundView

While working on #22 I was seeing a number of failures due to unused items in the file AudioGraphSoundView.js .
I commented them out to get it to pass lint, but it was a substantial part of the file, so I thought @jessegreenberg should review it.
The commit is 5a2976c. @jessegreenberg - can you please check this over and permanently remove the items if it is appropriate to do so?

Should the title words be included when searching or filtering the file list?

Under #36 we are adding the ability to search or filter the file list based on keywords. So far, we have talked about the keywords being explicitly specified on the 2nd line of the file. Should we include the words in the title in the list of keywords? In other words, if the first two lines of the file look like this:

// Altitude Image
// Keywords: image, model, control

...and the user entered the search term "altitude", should it match?

Investigating interface needs for the papyrus tool

Currently, we have four required tabs to use the tool. The camera, the projector, the editor (opened from camera), and the new board.html. We would like to identify the necessary features for papyrus and investigate how to combine or modify the existing interface to make using the tool more efficient (and aesthetic if possible).

Creating this as a broad issue to propose, discuss, and commit against.

@jbphet already took a step by modifying index.html to contain links of the needed HTML pages (8337519)

Add feature to lock down edits to Programs in specific playspaces

With the number of people now using and editing Programs in the remote database, we should be able to "lock down" edits on certain playspaces to prevent accidental overwrite/deletion. Another part of this safety measure is being implemented in #33 for backups.

We want to be able to prevent users from easily deleting, moving, or overwriting Programs in at least the templates playspace.

@jbphet said he had a few ideas. One including read-only access to those playspaces for non-admins. This is a little tricky since playspaces aren't "real", other than being a label in a column beside the Program code and number. Also, individuals could edit the programs in the repo (when #33 is done) and push, avoiding the remote admin requirement.

Not an idea we discussed, but: Might we also consider just adding a "make read only" button to the editor to indicate that a program is complete and should not be edited further without further consideration?

Synchronize the remote database to the repo

We want to be able to open up the Program database to contributions that take advantage of GitHubs security and contribution features. This will synchronize periodically with the database.

Assign after we have the database structured settled.

Get lint running

We would like lint to work in this repo. There is some lint support form the original project, but when we tried to run it it didn't work. We could figure out why. But even better, we should see if we can use PhET's lint rules in this repo.

Use more conventional method definitions in `CameraMain.js`

We are currently changing some functionality and adding new things to CameraMain.js, but there are some odd-looking method definitions. Currently, most of them look like this:

  _pollSpaceUrl = () => {
    .
    .
    .

This is unusual, at least in my experience. Usually, they look like:

  _pollSpaceUrl() {
    .
    .
    .

I'd like to fix this up, since we're going to be doing a fair amount of work in here.

Get the list of available scripts from the database

In the course of working on #5, a select element (aka a combo box) was added to allow users to choose a "space" from which to load and change shared programs. The list of spaces is currently hardcoded. This should be changed such that the list is obtained from the DB.

A way to output the state of the detected programs

(e.g., a list of programs and their current important code/variables (i.e., not the setup code, but the custom code), also listing what they are associated. I think ideally if a description sound scenario was created, we could archive or port that information easily to a design/sim code context)

I think this is nice for sharing to other designers/developers. Effectively capturing the state of the program the user created

Populate the database with templates and examples

We'd like to start creating as many general templates as we can to make it easier to find reusable Programs and have something for designers to pull from. This post contains a list of general templates we would like to create in the templates playspace.

Please recommend additional Programs in this issue and they will be added to the top post (this one). If you have access, you may also directly edit this post. Mark off templates that have been created.

  • Create Dial mapped to paper rotation
  • Create Radio Button
  • Create Slider with value mapped to paper position
    • And rotation
  • Use mouse click event to make something happen
    • Mouse down
    • Mouse up
    • Use mouse position to make something happen
  • Use keypress event to make something happen
    • Key down
    • Key Up
  • Create a model Property
  • Link to a model Property
  • Play sound (complete)
    • Play looping sound
    • Play sound in response to new value (e.g., from model Property)
  • Play TTS
    • Play TTS in response to new value (e.g., from model Property)
  • Create Image
    • Animate Image
  • View code for a slider that allows for Qualitative values
    • Specifically, wondering how a designer can design and test parameters for a slider.

Some more from 3/7/23:

  • Properties that implement linear, exponential, and other changes. Directly controlled by paper position.
  • Description mappers (value -> string, enumeration -> string, bool -> string)

What can we do to prepare papyrus for open source contributions?

@emily-phet and I discussed the opportunity here to make the repo friendly to open-source contributors. This is based on an open-source project and based on our other discussions about PhET in general, this seems like a manageable piece to put into practice our goals of creating a welcoming open-source community. If we can do this and point to features we'd like to see developed but are unable to commit resources to, then that is a bonus to the whole endeavor.

Using opensource.guide as a... guide. I've started building a list of items we should accomplish to transition the repo:

TODO to make this repo open-source friendly

GitHub-specific:

  • Make repo welcoming

Codebase: (these require issues of their own for devs to progress)

  • What can we clean up in the repo that we don't need?
  • Commenting as though someone not in PhET is definitely going to edit this
  • Maintaining ability to run the tool without reliance on PhET resources (that aren't already packaged with the tool) or hosted services. This includes being able to run a local database if desired.
  • Synchronize program database with repo. Or at least the "approved templates" we intend to create.

Non-GitHub Community/Documentation

  • Advertisement & solicitation (collaborators, web communities, etc)
  • CU webpage as option for friendly introduction and then links to documents for devs (APIs, etc)
    • Consider rewording of Contributing and ways to contribute
    • Consider FAQ
    • Consider resources to do dev/web dev

Scripts restart each time program is recognized even while still in window

Can we have scripts not completely restart when they lose detection? In the scripts themselves (pick-up/resume points..?) or in the detection algorithm?

JB via BF: Does each program call a new instance? Can you have the same program twice? This would assist with a β€œresume” function
Answer: Not at the moment. Can tear paper apart and it continues to recognize it

Image

Investigate flexibility of tool with phetlib integration at the script level

Discussed during meeting,

We would like to bias the flexibility of the tool in using our existing PhET libraries without embedding new functionality or abstraction into the tool code. Yes, this will make the scripts more complex, but less development will be needed on the tool side and more focus on the script side (which can involve everyone). We hope to generate enough example scripts that 1.) there's something to lean on to create new scripts and 2.) there are sufficient examples to modify on the fly, or even 3.) enough examples exist that someone might just "plug n play" from existing scripts.

I'm pairing that idea with our need to communicate values, variables, etc between scripts (e.g., a model environment script gets the mass value from another script).

Assigning JG and JB to flesh out details, questions, and needs.

React changes needed to modify the interface

We've been making good use of the Preview feature in the camera interface and it would be nice to have more area to play with the preview scripts (might want to rename this...) at typical laptop/monitor window aspect ratios/absolute sizes. Let's rearrange the the sidebar to exist below the camera output. I believe this is just some futzing with the CSS (move a few sections myself without loss of functionality) to change the Flexible Box setup.

I'll investigate myself a bit before assigning.

Support creating a model with paper programs

We started to look at how to create a model with paper programs. Our first pass encodes the model like this in a program:

  await paper.set( 'data', {
    model: {
      flyingThingAltitudeRange: {
        type: 'dot.Range',
        args: [ 0, 1 ]
      },
      flyingThingAltitudeProperty: {
        type: 'axon.NumberProperty',
        args: [ 0, 'model.flyingThingAltitudeRange' ]
      },

      // Time whenever the program is first detected, so internal code knows it is time to recreate this model
      updateTime: Date.now()
    }
  } );

Each model component has a named key, a type that refers to the component to be created from the phet-lib namespace, and args that hold the arguments for the component constructor.

Within paper programs code, we parse this and a single model from the data. The model itself is observable so that whenever it changes, view and controller components can update/reconstruct/delete.

In this example, the range is meant to be passed to the NumberProperty. This is not set up yet, but is something to figure out.

In team discussions we talked about having many papers to build up a model. But that seems very complicated. Having a single model paper is a good way to start and also may be the best way to construct a model with this tool. We will continue to prototype.

In other programs, you might be able to observe or control the model by referencing these keys.

Do we need to continue to support the standalone DB case?

For #5 we have changed the code to work with a remote DB that has a set of available spaces. These changes are likely to have impacted the ability of papyrus to work correctly if it is set up using a local DB, which is how the original Paper Programs code worked. Do we need to continue to support the use case where a local DB is set up and used to store the programs? If so, we will need to regression test this periodically and keep it working.

Is it okay to remove code from the printed papers permanently?

While working on #22, I came across an issue in printPdf.js where there was an unused variable called code in the function drawPage. I needed to remove it to pass lint, but it appears that the usage of this variable was commented out by us due to some issues with printing the code on the paper programs. I don't feel comfortable taking it out permanently without checking in with the rest of the team. Is it okay for us to fully remove, at least for now, the ability to print the code on the program sheets?

Update the Hello World template

While we plan to change the behavior of Create New Program so that it does not default create one from the Hello World template, it would be nice to have a thoroughly documented template Program that details what each section of the Program is intended for and our intended conventions (Line 1, keywords, async, Board, then Projector code)

Once we have keywords implemented (#36), I can probably draft this.

Do we need the "Original Code" column in the DB?

Is this crucial to their creation? It will always be the "stock template", I believe, which is hardcoded.

Can we replace it with "last save" instead so it is a one-instance version control?

Aside, and probably for its own issue: We can probably spin off the above idea to let it record the past e.g., 5 version or let us "lock" in one version of a program's code to protect against accidental overwrites or breaking changes.

How can we move Programs between Playspaces?

We're starting to create enough Programs in enough playspaces that there's a need for some more organization, especially now that we can easily duplicate Programs. It'd also be nice to be able to move Programs that are being tested in one playspace, more officially to another such as templates.

I tried an SQL query in ElephantSQL but was met with an error.

UPDATE programs
SET spaceName = 'newSpaceName'
WHERE number = [programNumber];

@jbphet said he'd investigate!

We might also consider adding a UI component to move a Program from one playspace to another. Maybe similar to the New Program dialog?

Let's figure out the right command to move Programs, then decide if it's frustrating enough to need a UI component.

Can the repo size be reduced?

@jessegreenberg and I just set up this fork and we each cloned it, and it's quite large. A du -sh papyrus command says that it's 166M, whereas many of our sim repos are around 8M. Can this be made smaller? @jessegreenberg says that the www directory that is included may be a build artifact that we could omit. If not that, perhaps some other things can go.

Set up a shared DB for storing the scripts

The paper programming environment uses a database for storing the scripts associated with each piece of paper. In our early prototyping, we've had each user set up a local instance of PostgreSQL. It would be good to migrate to a shared DB that is accessible from anywhere, but is also secure so that malicious actors can't mess it up.

@mattpen - You've got the most experience with SQL databases on the PhET team. How would you recommend starting to approach this?

Make Programs taggable with keywords and add interface search/filter by keyword in database

While we have plans to organize sets of Program examples by Playspace, we'd like to be able to filter the Programs by categories.

Currently, the Programs are named based on a comment written on Line 1. It was suggested that these keywords be added as the second comment line (Line 2) of the Program. Possibly comma separated or whatever is best for reading it back out with an SQL query.

To Dos

Database

  • Add the keyword tagging ability and have it updated in the database (a new column?)
  • Update our current examples

Interface

  • Spend the time to create a simple interface search/filter to only display Programs that contain the desired keywords

Linking this to #25 to determine where this fits into the interface as a whole. We can probably put this into the Editor page for now.

We'll also (later on) need to decide how this behaves with #34, which may let us reach across Playspaces without leaving the current one. Depending on how that is implemented, we might want to filter whatever view we have into the "remote" Playspace. However, this is not slated currently for implementation.

Add support for Voicing

We want the board Display to be able to speak with SpeechSynthesis and the Voicing feature. It should just need to be initialized where we create the Display.

Organization and access to central program repository (delete, search, duplicate, print)

We'll want to be able to do more to the database than is currently possible in Paper Programs.

@jessegreenberg and @jbphet successfully connected a hosted database to the tool and are investigating how to expose and give more access to the local spaces (so we could have more than one "repo" of scripts with different purposes or for different audiences)

We will definitely want convenient ways to Delete, Search for, duplicate, and some organizational method for versioning scripts.

We might consider creating a "Template" category or space where the most general examples go, with any necessary subcategories (model (maybe subcategory again: physics models, etc) , speech display, UI component).

Could the Templates go in their own space? Or just have their own label?

Can we restrict editing access to Templates, but leave duplicating and editing those open without restriction?

@jessegreenberg noted that when you are using a shared DB, the tool already locks out users who are editing a script. Two connections cannot edit the same script at the same time.

Description Design Flow Example

From https://docs.google.com/document/d/124JSjh6W3tCYkppOiXLZE4CFgyx-rXR9ArEwpV6lCX0/edit

@terracoda wrote:

I want to create an interactive that allows me to create and test a qualitative scale.
For example, if I want to test a qualitative scale for altitude or velocity. How would I go about doing that?
I am wondering how it would be different than for a scale designed for a slider, that I am directly controlling.
The first example is a scale for a changing context, whereas a slider is a scale for a changing value on an object that I am controlling.

@jessegreenberg wrote a similar:

I want to create an interactive that lets me explore an object response for one value and a context controlled by a second value. Changing the value for the object response triggers a value change of the context response.
Create program 1 - I declare it is a value for a slider.
Create program 2 - I declare it is a slider. I create a relationship between the slider and the value so that the slider controls the value.
Create program 3 - I declare that this is a collection of descriptions for the value of program 1. The program maps the range of values for program 1 to descriptions.
Create program 4 - I declare it is a value with a range. I declare it has a relation to program 1 such that the value of program 1 sets the value of program 4.
Create program 5 - I declare that this is a collection of descriptions for the value of program 5. The program maps the range of values for program 4 to descriptions.
I assign the value of program 3 to the object response of program 2.
I assign the value of program 5 to the object response of program 2.
When I move the slider in the scenery display, I hear the object and context response associated with that change.

It'd be nice to create a set of example scripts to enable this kind of user flow.

Enable paper overlap/proximity for use in Programs (Paper Programs Whisker functionality)

The Paper API from the legacy tool lets us use both the marker positions AND whether Programs (paper, not currently preview) are "touching".

https://github.com/janpaul123/paperprograms/blob/master/docs/api.md

Quoting from the API:

You can react to different events with a whisker

whisker.on('paperAdded', ({paperNumber, paper}) => {
console.log('added paper', paperNumber)
})

whisker.on('paperRemoved', ({paperNumber, paper}) => {
console.log('removed paper', paperNumber)
})

whisker.on('movedWhisker', ({ x, y }) => {
console.log('whisker tip x : ' + x + ' y: ' + y);
})
You can customize a whisker:

const whisker = paper.get('whisker', {
direction, // "up" (default), "down", "left", "right"
whiskerLength, // as fraction of the side (default 0.7)
requiredData, // array of data fields that must be present in the other paper
paperNumber, // paper number to do this for (default is own paper number)
color, // color of the whisker (default "rgb(255, 0, 0)")
})

It lets us do "something" when the paper begin overlap, stop overlap, or move relative to each other.

It lets us define required data that one program looks at from another to enable the behavior. This can set up a kind of hierarchy between programs.

Create optional program info on the Board

This has come up a few times in design meetings. It would be nice to draw some information about detected programs right in the board view. Displaying similar info to the current projector view. Probably conditionally and only displayed when a checkbox is checked. Debugging info would include a rectangle around the program and possibly the program number or title.

Create a tool-specific version of console.log

In development meeting today, we decided it would be helpful to have a log present on the Board that behaves like console.log. This would let individuals making Programs make debugging notes (and other possible use cases) that were directly visible on the Board without opening up the developer console.

@jbphet noted we could not intercede and copy the developer console (present in the browser developer tools, F12 on windows). We will need to create an analog of console.log. The devs indicated they would also likely still use console.log for transient debugging that designers/users would not necessarily need to see.

The devs also indicated it'd be best to put this behind a query parameter. When implemented, we should have a link to the board with and without this on the landing page (index.html)

Let's go ahead and implement this!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.