Coder Social home page Coder Social logo

pod-cached's Introduction

Pod::To::Cached

Create and Maintain a cache of precompiled pod files

Module to take a collection of pod files and create a precompiled cache. Methods / functions to add a pod file to a cache.

Install

This module is in the Perl 6 ecosystem, so you install it in the usual way:

zef install Pod::To::Cached

SYNOPSIS

use Pod::To::Cached;

my Pod::To::Cached $cache .= new(:path<path-to-cache>, :source<path-to-directory-with-pod-files>);

$cache.update-cache;

for $cache.hash-files.kv -> $source-name, $status {
    given $status {
        when 'Current' {say "$source-name」 is up to date with POD source"}
        when 'Valid' {say "$source-name」 has valid POD, but newer POD source contains invalid POD"}
        when 'Failed' {say "$source-name」 is not in cache, and source file contains invalid POD"}
        when 'New' { say "$source-name」 is not in cache and cache has not been updated"}
        when 'Old' { say "$source-name」 is in cache, but has no associated pod file in DOC"}
    }
    user-supplied-routine-for-processing-pod( $cache.pod( $source-name ) );
}

# Find files with status
say 'These pod files failed:';
.say for $cache.list-files( 'Failed' );
say 'These sources have valid pod:';
.say for $cache.list-files(<Current Valid>);

# Find date when pod added to cache
my $source = 'language/pod'; # name of a documentation source
say "$source」 was added on 「{ $cache.cache-timestamp( $source ) }";

# Remove the dependence on the pod source
$cache.freeze;

Notes

  • Str $!path = '.pod6-cache'
    path to the directory where the cache will be created/kept

  • Str $!source = 'doc'
    path to the collection of pod files ignored if cache frozen

  • @!extensions =
    the possible extensions for a POD file

  • verbose = False
    Whether processing information is sent to stderr.

  • new
    Instantiates class. On instantiation,

    • get the cache from index, or creates a new cache if one does not exist
    • if frozen, does not check the source directory
    • if not frozen, or new cache being created, verifies
      • source directory exists
      • the source directory contains POD/POD6 etc files (recursively)
      • no duplicate pod file names exist, eg. xx.pod & xx.pod6
    • verifies whether the cache is valid
  • update-cache
    All files with a modified timestamp (reported by the filesystem) after the added instant are precompiled and added to the cache

    • Status is changed to Updated (compiles Valid) or Fail (does not compile)
    • Failed files that were previously Valid files still retain the old cache handle
    • Throws an exception when called on a frozen cache
  • freeze
    Can be called only when there are only Valid or Updated (no New, Tainted or Failed files), otherwise dies.
    The intent of this method is to allow the pod-cache to be copied without the original pod-files.
    update-cache will throw an error if used on a frozen cache

  • list-files( Str $s --> Positional ) returns an Sequence of files with the given status

  • list-files( Str $s1, $s2 --> Positional ) returns an Array of files with the given status list

  • hash-files( *@statuses? --> Associative ) returns a map of the source-name and its statuses

    • explicitly give required status strings: C<< $cache.hash-files() >>
    • return all files C< $cache.hash-files >
  • cache-timestamp( $source --> Instant ) returns the Instant when a valid version of the Pod was added to the cache

    • if the time-stamp is before the time the Pod was modified, then the pod has errors
    • a Failed source has a timestamp of zero
  • pod

    • method pod(Str $source)
    • Returns an array of POD Objects generated from the file associated with $source name.
    • When a doc-set is being actively updated, then pod files may have failed, in which case they have Status Valid.
    • To froze a cache, all files must have Current status
  • Status is an enum with the following elements and semantics

    • Current
      There is a compiled source in the cache with an added date after the modified date
    • Valid
      There is a compiled source in the cache with an added date before the modified date and there has been an attempt to add the source to cache that did not compile
    • Failed
      There is not a compiled source in the cache, but there has been an attempt to add the source name to the cache that did not compile
    • New
      A new pod source has been detected that is not in cache, but C has not yet been called to compile the source. A transitional Status
    • Old
      A source name that is in the cache but no longer reflects an existing source.

LICENSE

You can use and distribute this module under the terms of the The Artistic License 2.0. See the LICENSE file included in this distribution for complete details.

The META6.json file of this distribution may be distributed and modified without restrictions or attribution.

pod-cached's People

Contributors

antoniogamiz avatar finanalyst avatar jj avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar

Forkers

jj

pod-cached's Issues

.update-cache functionality

In order to only regenerate those files who have changed, I need to know the name of the files. Currently, I'm getting them parsing the output that .update-cache produces. This is a kind of a hack so it would be useful if .update-cache returns those names.

Support for several source dirs

Problem

At this moment, we can only specify a single source directory with :source. My idea convert :source to an array.

Why do I want this? Perl6::Documentable supports pods inside files containing code (.p6, .pm6, etc), so in order to use them, we would need to specify `:source("lib"), to gather all documentation in our module.

In that case we face two problems:

  • Pod::To::Cached does not support those extensions (I suppose that's easy to fix)
  • We have already used :source. What if we want to create an additional directory containing tutorials in pod6? We would need to use :source("<module-root-directory>").

The last fix is a hack I don't like because you can have pods in your module root directory, which you do not want to be used by Perl6::Documentable.

Suggestion

  • Add support for .p6 and .pm6 files.
  • Add support for :@source.

This is only my idea, please let me know your opinion.

Module name

This module is listed as Pod::To::Cached (which would make sense, BTW, if we were going to do perl6 --doc=Cached) but actually it installs Pod::Cached. Is there some refactoring missing here?

Use enum for status

Right now it's using strings, which is convenient, but it would be more type safe if we used an enum, with the 4 possible statuses.

Add heavy-duty 256 files test

Some problems arise when many files are open, maybe in a threaded environment. My hunch is that the intermitent failure is related to this. If we could golf it down to a test here, that would be great. And if it does not fail, it will prove that it works well and the problem might be in the threads or somewhere else.

Flapper: it fails some times

When launching it from documentable, it freezes with some random files, and sometime fails with;

An operation first awaited:
  in method update-cache at /home/docs-stage.perl6.org/.rakudobrew/versions/moar-2019.07.1/install/share/perl6/site/sources/D8380CC32DB7035B76CD2282A9EF7BCE5986F884 (Pod::To::Cached) line 262
  in submethod BUILD at /home/docs-stage.perl6.org/.rakudobrew/versions/moar-2019.07.1/install/share/perl6/site/sources/9562BE19FD1AFF14C1E2B76BC6DA2A8CC2737B8F (Documentable::Registry) line 62
  in sub MAIN at /home/docs-stage.perl6.org/.rakudobrew/versions/moar-2019.07.1/install/share/perl6/site/sources/6247B66BE996BA5368912546E42DEFC230ED939F (Documentable::CLI) line 115
  in sub RUN-MAIN at /home/docs-stage.perl6.org/.rakudobrew/versions/moar-2019.07.1/install/share/perl6/site/sources/6247B66BE996BA5368912546E42DEFC230ED939F (Documentable::CLI) line 21
  in block <unit> at /home/docs-stage.perl6.org/.rakudobrew/versions/moar-2019.07.1/install/share/perl6/site/resources/BCB7763E5CDF2EC0405E0F05D9F01F7C87E49CF1 line 3
  in sub MAIN at /home/docs-stage.perl6.org/.rakudobrew/versions/moar-2019.07.1/install/share/perl6/site/bin/documentable line 3
  in block <unit> at /home/docs-stage.perl6.org/.rakudobrew/versions/moar-2019.07.1/install/share/perl6/site/bin/documentable line 1

Died with the exception:
    No such method 'list' for invocant of type 'Mu'. Did you mean any of these?
        gist
        isa

      in any  at /home/docs-stage.perl6.org/.rakudobrew/versions/moar-2019.07.1/install/share/perl6/runtime/CORE.d.setting.moarvm line 1
      in method update-cache at /home/docs-stage.perl6.org/.rakudobrew/versions/moar-2019.07.1/install/share/perl6/site/sources/D8380CC32DB7035B76CD2282A9EF7BCE5986F884 (Pod::To::Cached) line 262
      in submethod BUILD at /home/docs-stage.perl6.org/.rakudobrew/versions/moar-2019.07.1/install/share/perl6/site/sources/9562BE19FD1AFF14C1E2B76BC6DA2A8CC2737B8F (Documentable::Registry) line 62
      in sub MAIN at /home/docs-stage.perl6.org/.rakudobrew/versions/moar-2019.07.1/install/share/perl6/site/sources/6247B66BE996BA5368912546E42DEFC230ED939F (Documentable::CLI) line 115
      in sub RUN-MAIN at /home/docs-stage.perl6.org/.rakudobrew/versions/moar-2019.07.1/install/share/perl6/site/sources/6247B66BE996BA5368912546E42DEFC230ED939F (Documentable::CLI) line 21
      in block <unit> at /home/docs-stage.perl6.org/.rakudobrew/versions/moar-2019.07.1/install/share/perl6/site/resources/BCB7763E5CDF2EC0405E0F05D9F01F7C87E49CF1 line 3
      in sub MAIN at /home/docs-stage.perl6.org/.rakudobrew/versions/moar-2019.07.1/install/share/perl6/site/bin/documentable line 3
      in block <unit> at /home/docs-stage.perl6.org/.rakudobrew/versions/moar-2019.07.1/install/share/perl6/site/bin/documentable line 1

Use specific exceptions classes

I am trying to handle exceptions, but they are AdHoc so I do not have an easy way to catch them separately. It would be useful to have specific types for each case.

Not really a test error, but a warning

But it says this when runnint 030-sub-directories.t

Testing started at 18:46 ...
WARNING: unhandled Failure detected in DESTROY. If you meant to ignore it, you can mark it as handled by calling .Bool, .so, .not, or .defined methods. The Failure was:
Failed to open file /home/jmerelo/Code/forks/raku/pod-cached/t/tmp/doc/pod-file-to-deprecate.pod6: No such file or directory
  in block  at /home/jmerelo/Code/forks/raku/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 160
  in method compile at /home/jmerelo/Code/forks/raku/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 153
  in method update-cache at /home/jmerelo/Code/forks/raku/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 129
  in block <unit> at t/030-sub-directories.t line 29
Pod::To::Cached.new(path => "t/tmp/ref", source => "t/doctest", verbose => Bool::False, precomp => CompUnit::PrecompilationRepository::Document.new(store => CompUnit::PrecompilationStore::File.new(prefix => IO::Path.new("t/tmp/ref", :SPEC(IO::Spec::Unix), :CWD("/home/jmerelo/Code/forks/raku/pod-cached")))), files => {:community(${:added(Instant.from-posix(<607771848895/384>, Bool::False)), :cache-key("418B03C91215CC919B0411338E64A1F9A8C01E9A"), :handle(CompUnit::Handle.new), :path("t/doctest/community.pod6"), :status(Status::Current)}), :contexts(${:added(Instant.from-posix(<1134823998751/717>, Bool::False)), :cache-key("14700E97A4DA0BE29B50C0DAE3132B7C369E29C6"), :handle(CompUnit::Handle.new), :path("t/doctest/contexts.pod6"), :status(Status::Current)}), :operators(${:added(Instant.from-posix(<848348205853/536>, Bool::False)), :cache-key("697823D308688F7C73EC9A1160ACC4FC207EA540"), :handle(CompUnit::Handle.new), :path("t/doctest/operators.pod6"), :status(Status::Current)}), :simple(${:added(Instant.from-posix(<2527634485744/1597>, Bool::False)), :cache-key("0F7D0D088B6EA936FB25B477722D734706FE8B40"), :handle(CompUnit::Handle.new), :path("t/doctest/simple.pod6"), :status(Status::Current)}), ("sub/simple") => ${:added(Instant.from-posix(<1046190604230/661>, Bool::False)), :cache-key("316124F172D0B93596116FC65D771ACB920018E0"), :handle(CompUnit::Handle.new), :path("t/doctest/sub/simple.pod6"), :status(Status::Current)}}, frozen => Bool::False, error-messages => Array[Str].new())

The file mentioned is created in the previous test, but it should disappear, since we're deleting the cache. It does not.

Installation fails on Windows

Steps to reproduce:

  • Download and install Rakudo Star 2019.03
  • zef upgrade
  • zef update
  • zef install Pod::Cached
===> Searching for: Pod::Cached
No candidates found matching identity: Pod::Cached
  • git clone https://github.com/finanalyst/pod-cached
  • zef install ./pod-cached

Error output:

> zef install ./pod-cached/
===> Testing: Pod::To::Cached:ver<0.3.0>
# Failed test 'doc cache created with sub-directories'
# at t\030-sub-directories.t line 25
# $!path has corrupt doc-cache
Cannot look up attributes in a Pod::To::Cached type object
  in method update-cache at C:\Users\tester\perl6\pod-cached\lib\Pod\To\Cached.pm6 (Pod::To::Cached) line 240
  in block <unit> at t\030-sub-directories.t line 26

# test pod extraction
$!path has corrupt doc-cache
  in submethod TWEAK at C:\Users\tester\perl6\pod-cached\lib\Pod\To\Cached.pm6 (Pod::To::Cached) line 146
  in block <unit> at t\040-pod-extraction.t line 16

# Heavy duty test optional
Aborting due to test failure: Pod::To::Cached:ver<0.3.0> (use --force-test to override)
===> Testing [FAIL]: Pod::To::Cached:ver<0.3.0>

Desktop:

  • OS: Windows 10 x64
  • Version 10.0.18362
  • Rakudo Star 2019.03

Related issue:

Extracting pod from a document

You have probably seen Raku/Pod-To-HTML#55, which is crucial for making pod6 be rendered in GitHub. As a prerequisite, we need a new way of obtaining the pod of a document without using eval. Do you think you could spin off CompUnit::PrecompilationRepository::Document for that and use it to simply cache and obtain the pod for a document?

Current version fails tests with 2019.03.1

This was working a short time ago, all of a sudden it's failed.

===> Testing: Pod::To::Cached:ver<0.3.0>
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of uninitialized value of type Status in string context.
Methods .^name, .perl, .gist, or .say can be used to stringify it to something meaningful.
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 369
# Failed test 'One current, one tainted'
# at t/020-source.t line 137
# expected: ${:a-pod-file("Current"), :a-second-pod-file("Valid")}
#      got: ${:a-pod-file(""), :a-second-pod-file("Valid")}
# Failed test 'List with list of statuses'
# at t/020-source.t line 139
# expected: $("a-pod-file", "a-second-pod-file")
#      got: $("a-second-pod-file",)
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of uninitialized value of type Status in string context.
Methods .^name, .perl, .gist, or .say can be used to stringify it to something meaningful.
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 369
# Failed test 'One current, one tainted'
# at t/020-source.t line 156
# expected: ${:a-pod-file("Valid"), :a-second-pod-file("Current")}
#      got: ${:a-pod-file("Valid"), :a-second-pod-file("")}
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
# Looks like you failed 3 tests of 36
# test pod extraction
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Use of Nil in string context
  in block  at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 170
Attempt to obtain non-existent POD for <a-pod-file>. Is the source new and failed to compile? Has the cache been updated?
  in method pod at /home/jmerelo/progs/forks/perl6/pod-cached/lib/Pod/To/Cached.pm6 (Pod::To::Cached) line 344
  in block <unit> at t/040-pod-extraction.t line 18

# Heavy duty test optional
===> Testing [FAIL]: Pod::To::Cached:ver<0.3.0>
Aborting due to test failure: Pod::To::Cached:ver<0.3.0> (use --force-test to override)

Can't install module

$ perl6 --version
This is Rakudo version 2019.07.1-239-g579ac66e8 built on MoarVM version 2019.07.1-134-ga37504d12
implementing Perl 6.d.
$ zef install Pod::To::Cached
===> Searching for: Pod::To::Cached
===> Testing: Pod::To::Cached:ver<0.3.3>
# Failed test 'One current, one tainted'
# at t/020-source.t line 137
# expected: ${:a-pod-file("Current"), :a-second-pod-file("Valid")}
#      got: ${:a-pod-file("Current"), :a-second-pod-file("Current")}
# Failed test 'One current, one tainted'
# at t/020-source.t line 156
# expected: ${:a-pod-file("Valid"), :a-second-pod-file("Current")}
#      got: ${:a-pod-file("Current"), :a-second-pod-file("Current")}
# Failed test 'One remains Valid because new version did not compile'
# at t/020-source.t line 159
# expected: ${:a-pod-file("Valid"), :a-second-pod-file("Current")}
#      got: ${:a-pod-file("Current"), :a-second-pod-file("Current")}
# Failed test 'Error detected in pod'
# at t/020-source.t line 161
# expected a match with: /'Compile error'/
#                   got: ""
# Failed test 'new file not added to cache because it did not compile'
# at t/020-source.t line 168
# Failed test 'hash-files with seq of statuses correct'
# at t/020-source.t line 221
# expected: ${:a-pod-file("Valid"), :pod-file-to-deprecate("Old")}
#      got: ${:pod-file-to-deprecate("Old")}
# You failed 6 tests of 36
# test pod extraction
# testing freeze
# Heavy duty test optional
===> Testing [FAIL]: Pod::To::Cached:ver<0.3.3>
Aborting due to test failure: Pod::To::Cached:ver<0.3.3> (use --force-test to override)

JSON::Fast is also missing from META6.json

But this is also used by Pod::Render, which as I said in #2 should probably be left as an example, rather than included. It's not even included in the "provides" section of META6.json...

If files are moved and the cache exists, it hangs up

If you change the structure of a cached folder and don't delete the cache. it will hang up.

To reproduce:

  • Cache one directory foo with files bar baz
  • mkdir foo/quux
  • cd foo
  • mv bar baz quux

Try to run the program again... It simply freezes.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.