drewolson / scrivener Goto Github PK

I've detailed the problem at the Scrivener Elastic project [here] (radar/scrivener_elastic#1)

Basically I want to add a new field to the Config struct. If it's not set, it will return all the pages found( can be a large number ie 1_000_000). If it's set then it will show only the first max_number_of_pages pages.
Can we add this field it to Scrivener.Config?

I've also noticed this open issue that is a bit similar to my situation but it's for Scrivener Ecto use case. Maybe we can find a way to avoid this types of issues in the future, since you need to change the Scrivener.Config every time you need a new field for a package that just happens to use Scrivener.
Maybe a :options/:custom_options field that can hold custom parameters will do the trick. In this way we won't need to create extra fields in Scrivener for each use case and keep the package as slim and general as possible.

Hey I know in 2.20 :options were introduced, but I was just wondering how I would go about actually using them? I want to limit the total number of pages returned for queries that can return millions of results. I've tried looking at the hexdocs but still can't figure it out.

One of the benefits of scrivener is being able to do Repo.paginate(query, params) where params comes from a user input (in the case of Phoenix.) Currently it is possible for a user to set page_size to a high value without restriction.

There are a few solutions I can think of outside of Scrivener:

• A function that you can call before paginate which looks for a page_size parameter and sets it to a defined max value if the page_size exceeds this value
• A plug which does the same as above
• Add a Repo.paginate_with_limit function which does the same thing and calls paginate

None of these options are quite as convenient as just passing through params though. I was wondering if it is justified to have a max_page_size config option? It could default to :infinity which make the pagination behave as it does today.

Hey man! Hope things are good!

I'm running a local fork of Scrivener/Scrivener_Ecto right now that adds a show_totals: boolean field to the Config struct.

The default is true, since I only turn it off for certain tables.

The use case is when working with unbounded data (like a sensor reading table in postgres, or gps points, etc), calculating the totals can cause DB timeouts.

In particular, I think pagination has really nice use cases w/ the new Flow module for processing large sets of data.

If you like the idea, I can submit a PR w/ tests to the repos monday! Let me know. Scrivener is solid by the way, thanks for creating it!

When trying to use the library I get

no function clause matching in Scrivener.HTML.pagination_links/2

my html looks like this

<div class="col-sm-12">
    <h3 class="text-center">Recent Auctions</h3>
    <%= for auction <- @recent_auctions do %>
      <%= render "auction_block.html", auction: auction %>
    <% end %>
    <%= pagination_links @conn, @page %>
  </div>

my controller looks like

query = from a in Auction,
             order_by: [desc: a.time],
             preload: [:state, :city, :facility]
      page = Repo.paginate(query, params)

      assign(conn, :user_city, nil)
      |> assign(:auctions, page.entries)
      |> assign(:page_number, page.page_number)
      |> assign(:page_size, page.page_size)
      |> assign(:total_pages, page.total_pages)
      |> assign(:total_entries, page.total_entries)
      |> assign(:recent_auctions, page.entries)
      |> assign(:page, page.entries)
      |> assign(:recent_auctions_for_user_city, [])

Any ideas what is going on?

The upgrade to new phoenix (1.1) fails with error:

Looking up alternatives for conflicting requirements on postgrex
Activated version: 0.9.1
From mix.exs: >= 0.0.0
From ecto v1.1.0: ~> 0.10
From scrivener v1.1.0: ~> 0.9.0

** (Mix) Hex dependency resolution failed, relax the version requirements or unlock dependencies

I have a case where the number of records is relatively large, so it produces e.g. 50+ pages. This cases a bit of an issue with UI since those pages don't fit the page nicely (break into multiple rows etc.).

For kaminari for example, this is configurable (even has default behavior) via various window related params. So, is there a way to achieve the same for scrivener?

If not, would there be interest in adding this to scrivener or do you prefer to leave it to the consumers?

Thank you for your time 😄

P.S. If there'd be interest in adopting this for scrivener, I have to solve this for a project, might ask the owner to devote some time to add this to scrivener so I might help.

What's the best way to use this with a preloaded association?

Something like?

page = from c in Comment, order_by: c.published_at |> Repo.paginate(params)
Repo.all from p in Post, preload: [comments: ^page.entries]

Thanks for the code.

Was attempting to use to paginate a logs table I have that I'm ordering by timestamp but otherwise don't keep an ID. Perhaps consider allowing a configurable count field?

Thanks

Hi, I'm new to Elixir so trying to add small features to applications right now. Please excuse me if this is totally dumb, but I've spent a while trying to debug and I'm lost.

I started to work on rrrene/elixirstatus-web#13 which was to add pagination using Scrivener. Here is what I've done: rrrene/elixirstatus-web@master...philnash:pagination. I hoped that it would be mostly as simple as adding Scrivener and changing the call to Repo.all to Repo.paginate in the PostingsController. However, I get the error Array type is not supported by MySQL. The full stack trace is below:

[error] #PID<0.490.0> running ElixirStatus.Endpoint terminated
Server: localhost:4000 (http)
Request: GET /postings
** (exit) an exception was raised:
    ** (ArgumentError) Array type is not supported by MySQL
        (ecto) lib/ecto/adapters/mysql/connection.ex:352: Ecto.Adapters.MySQL.Connection.expr/2
        (elixir) lib/enum.ex:1008: anonymous fn/4 in Enum.map_join/3
        (elixir) lib/enum.ex:1261: Enum."-reduce/3-lists^foldl/2-0-"/3
        (elixir) lib/enum.ex:1007: Enum.map_join/3
        (ecto) lib/ecto/adapters/mysql/connection.ex:347: Ecto.Adapters.MySQL.Connection.expr/2
        (elixir) lib/enum.ex:1008: anonymous fn/4 in Enum.map_join/3
        (elixir) lib/enum.ex:1261: Enum."-reduce/3-lists^foldl/2-0-"/3
        (elixir) lib/enum.ex:1007: Enum.map_join/3

I can understand MySQL not being able to do some things, but this seems like a simple use case and I can't tell where or how I am trying to use the Array type.

Any help would be gladly received. Thanks.

Problem:

Cannot pass a list of DB results to the Scrivener.paginate/2 function.

Description

Sometimes I find that I get a list of results from the DB using Ecto and I need to do a some operation on the list outside the realm of Ecto and Ecto Queries.
With the finalized list post processing, I still need to paginate the list but currently scrivener does not support passing in a list for pagination.

Proposed Solution:

Add ability to paginate a list.

I've started on this already and trying it out at the moment. I'd appreciate your feedback.

Thanks for writing this library BTW.

When in Add

use Scrivener, page_size: 10

to my repo I get the error

module Scrivener is not loaded could not be found, even after fetching the dependencies. I am using the latest phoenix for my project

The legal team at my company doesn't like us using repositories without license files in the repository itself. Would it be possible for you to add the MIT license file to the repo?

Since you already need to retrieve the count of entries to get the total number of pages, it would be beneficial to add it to the Page struct so the user doesn't need to retrieve it again

Hi @drewolson, I'm having issues with dialyzer when using latest available hex package:

:0:unknown_function
Function Scrivener.paginate/2 does not exist.
________________________________________________________________________________
:0:unknown_function
Function Scrivener.Config.new/3 does not exist.
________________________________________________________________________________
:0:unknown_type
Unknown type: Scrivener.Page.t/0.
________________________________________________________________________________
done (warnings were emitted)

If I switch to using scrivener and scrivener_ecto from github master, then it works fine.

Could you please release a new version with these type fixes? 🙂

Thanks! 👍

I had a similar problem than this guy.

I tried to make a query grouping by month, using fragment and date_trunc, it works with Repo.all() but with Repo.paginate() I get
expected at most one result but got 2 in query:

With this query:

    query = from u in User,
      join: r in assoc(u, :roles),
      where: not r.name in ["a", "b"],
      preload: [roles: r]

The sql returns 28 rows, but there's only 20 unique users.

%Scrivener.Page{entries: [...], page_number: 1, page_size: 10, total_entries: 28,
 total_pages: 3}

When I iterate through the items page 1 has 10 items, page 2 has 7 items, and page 3 has 3 items.

I'm pretty new to Ecto, so maybe my query is wrong, but it looks like scrivener isn't recognizing the duplicate rows from the join.

Would be great to have an option to don't show the pagination when there is only 1 page with results.

My Application's Repo looks like this

defmodule AppName.Repo do
  use Ecto.Repo, otp_app: :app_name
  use Scrivener, page_size: 20
end

My index action looks like this...

def index(conn, params) do
  words = from(w in Word, order_by: [asc: :abc_id], preload: [:languages]) 
    |> Repo.paginate(params)
  render(conn, "index.json",
    %{
      words: words.entries,
      current_page: words.page_number,
      total_pages: words.total_pages,
      per_page: words.page_size
    }
  )
end

Versions of other notable deps in mix file.

  scrivener 1.0.0 
  phoenix 1.0.2 
  phoenix_ecto 1.2.0
  ecto 1.0.3

P.S: Scrivener.paginate/2 works with %Scrivener.Config.

After upgrading from version 2.7.0 to version 2.7.1 we have the following compilation error:

Dependencies have diverged:
* ex_doc (Hex package)
  the :only option for dependency ex_doc

  > In mix.exs:
    {:ex_doc, "~> 0.22", [env: :prod, repo: "hexpm", hex: "ex_doc", only: :dev, runtime: false]}

  does not match the :only option calculated for

  > In deps/scrivener/mix.exs:
    {:ex_doc, ">= 0.0.0", [env: :prod, hex: "ex_doc", repo: "hexpm", optional: false]}

  Remove the :only restriction from your dep
** (Mix) Can't continue due to errors on dependencies

It seems it didn't like this change: https://diff.hex.pm/diff/scrivener/2.7.0..2.7.1#84732810-30-

Thank you for your time!

Hello, I had pagination working fine in my app using:

• phoenix 1.1.4
• phoenix_ecto 2.0
• scrivener_html 1.1
• scrivener 1.1.4

But now, after upgrading to:

• phoenix 1.2.0
• phoenix_ecto 3.0
• scrivener_html 1.3
• scrivener 2.0.0

It's failing with:

[error] #PID<0.4760.0> running MyApp.Endpoint terminated
Server: localhost:4000 (http)
Request: GET /s
** (exit) an exception was raised:
    ** (Protocol.UndefinedError) protocol Scrivener.Paginater not implemented for #Ecto.Query<from s in MyApp.Sitemap, where: s.user_id == ^1, order_by: [desc: s.inserted_at]>
        lib/scrivener/paginater.ex:1: Scrivener.Paginater.impl_for!/1
        lib/scrivener/paginater.ex:10: Scrivener.Paginater.paginate/2
        (MyApp) web/controllers/sitemap_controller.ex:28: MyApp.SitemapController.index/2
        (MyApp) web/controllers/sitemap_controller.ex:1: MyApp.SitemapController.action/2
        (MyApp) web/controllers/sitemap_controller.ex:1: MyApp.SitemapController.phoenix_controller_pipeline/2
        (MyApp) lib/MyApp/endpoint.ex:1: MyApp.Endpoint.instrument/4
        (MyApp) lib/phoenix/router.ex:261: MyApp.Router.dispatch/2
        (MyApp) web/router.ex:1: MyApp.Router.do_call/2
        (MyApp) lib/MyApp/endpoint.ex:1: MyApp.Endpoint.phoenix_pipeline/1
        (MyApp) lib/plug/debugger.ex:93: MyApp.Endpoint."call (overridable 3)"/2
        (MyApp) lib/MyApp/endpoint.ex:1: MyApp.Endpoint.call/2
        (plug) lib/plug/adapters/cowboy/handler.ex:15: Plug.Adapters.Cowboy.Handler.upgrade/4
        (cowboy) src/cowboy_protocol.erl:442: :cowboy_protocol.execute/4

I've already cleaned up and reinstalled the deps, not sure what is failing here. This is the code in the controller:

  def index(conn, params) do
    sitemaps = Sitemap
               |> where(user_id: ^(current_user(conn).id))
               |> order_by(desc: :inserted_at)
               |> Repo.paginate(params)

    render(conn, "index.html", sitemaps: sitemaps)
  end

Any idea?
Thanks!

(Comes from mgwidmann/scrivener_html#30)

The following statement works with 1.1.1

iex(1)> (from c in Comment, join: u in assoc(c, :user), order_by: [desc: u.inserted_at]) |> Repo.paginate(page: 1, page_size: 10)
[22:43:12.618] [debug]  SELECT count('*') FROM "comments" AS c0 INNER JOIN "users" AS u1 ON u1."id" = c0."user_id" [] OK query=41.0ms
[22:43:12.679] [debug]  SELECT c0."id", c0."content", c0."username", c0."email", c0."deleted", c0."approved", c0."inserted_at", c0."updated_at", c0."user_id", c0."post_id", c0."parent_id" FROM "comments" AS c0 INNER JOIN "users" AS u1 ON u1."id" = c0."user_id" ORDER BY u1."inserted_at" DESC LIMIT $1 OFFSET $2 [10, 0] OK query=60.7ms queue=0.1ms

but raises an error with 1.1.3

iex(4)> (from c in Comment, join: u in assoc(c, :user), order_by: [desc: u.inserted_at]) |> Repo.paginate(page: 1, page_size: 10)
[22:55:14.153] [debug]  SELECT count(DISTINCT c0."id") FROM "comments" AS c0 INNER JOIN "users" AS u1 ON u1."id" = c0."user_id" [] OK query=69.8ms
[22:55:14.160] [debug]  SELECT c0."id" FROM "comments" AS c0 INNER JOIN "users" AS u1 ON u1."id" = c0."user_id" GROUP BY c0."id" ORDER BY u1."inserted_at" DESC LIMIT $1 OFFSET $2 [10, 0] ERROR query=6.8ms queue=0.1ms
** (Postgrex.Error) ERROR (grouping_error): column "u1.inserted_at" must appear in the GROUP BY clause or be used in an aggregate function
         (ecto) lib/ecto/adapters/sql.ex:185: Ecto.Adapters.SQL.query!/5
         (ecto) lib/ecto/adapters/sql.ex:481: Ecto.Adapters.SQL.execute/6
         (ecto) lib/ecto/repo/queryable.ex:95: Ecto.Repo.Queryable.execute/5
         (ecto) lib/ecto/repo/queryable.ex:15: Ecto.Repo.Queryable.all/4
    (scrivener) lib/scrivener.ex:153: Scrivener.entries/4
    (scrivener) lib/scrivener.ex:102: Scrivener.paginate/2

Is there any plan to disable "select count(*)" by param?,
I use automatic pagination on my website, like facebook (when you research end of page, it load another one) and I'd like to not did 2 selects all the time.

my logic is:
get next page with 30 items

• if return exactly 30, maybe has a another page
• else isn't.

So, my client will make another ajax call only when last call returned exactly 30 items. It's a performace gain!

tip: I look you code, and you are calling total_entries before entries. You could test if entries returned less than page_size and just calculate total_entries. Will be 1 select less on last page ;)

Please let me know if this isn't the appropriate venue – or just close the issue :) – I also noticed that this project itself is in 'low maintenance mode', so no worries if this isn't something you'd be interested in in helping out with.

Scrivener.HTML seems to be un-maintained; see:

• Maintenance Help · Issue #103 · mgwidmann/scrivener_html

Several users of that library have contributed changes to resolve some issues, mainly due to dependency conflicts as various other libraries have been updated. I think most of the desired changes are literally just fiddling with dependency versions, e.g. relaxing version requirements.

Unfortunately, the owner of that project/package doesn't seem able or willing to follow-up on any of the open issues. (And that's totally okay!) But perhaps you, the creator/owner/maintainer and various contributors to this project, might be able to contact the creator/owner of Scrivener.HTML yourself and ask them to consider transferring ownership of the project to someone else. (And it's totally fine if no one has the time, or wants to spend it, to do any of that.)

Worst case, the users of this library can always fork it themselves – several have done so already.

Maybe second-best case would be someone stepping up to maintain a fork. That seems like it would also require changing the project/package name (e.g. on Hex).

Thanks for the project!

Having issues with one of my functions because I try to do .entries on a Scrivener map. Any ideas what the below means? Any help at all is absolutely appreciated.

          #{'entries' := __@1} can never match the type 

A little bit more info about what I'm doing is here. https://elixirforum.com/t/handle-adding-fields-into-map/31846

Hey,

We've stumbled upon an issue with Scrivener and joined models counting, which prevents correct pagination.

Here is how it works:

1. Have a model with many relationships
2. Try to load it paginated with many joins:
   (In the following, Article is our model with many relationships)

result = Article |> Article.with_tags_and_assets |> Repo.paginate(%{"page_size"=> 3})
pry(26)> result.entries |> Enum.count                                                                                      
1

# BUT
result = Article |> Article.with_tags_and_assets |> Repo.paginate(%{"page_size"=> 300})
pry(28)> result.entries |> Enum.count                                                                                        
3

# with_tags_and_assets is:

def with_tags_and_assets(q) do
  q
  |> join(:left, [a], t in assoc(a, :tags))
  |> join(:left, [a], i in assoc(a, :images))
  |> join(:left, [a], d in assoc(a, :documents))
  |> join(:left, [a], v in assoc(a, :videos))
  |> preload([a, t, i, d, v], [tags: t, images: i, documents: d, videos: v])
end

# And the relationships definitions in Article

has_many :article_tags, MyApp.Models.ArticleTag
has_many :tags, through: [:article_tags, :tag]

has_many :article_assets, MyApp.Models.ArticleAsset
has_many :images, through: [:article_assets, :image]
has_many :documents, through: [:article_assets, :document]
has_many :videos, through: [:article_assets, :video]

This is caused by a mismatch between the number of rows returned by the DB, and the actual number of entities that you get, since with joins, 1 entity >= 1 row.

Especially:

• The counting method here: https://github.com/drewolson/scrivener/blob/master/lib/scrivener.ex#L152 can't work.
• The offset here is not computed correctly: https://github.com/drewolson/scrivener/blob/master/lib/scrivener.ex#L144

There is a SQL solution, presented here: http://stackoverflow.com/questions/15897055/proper-pagination-in-a-join-select/15897271#15897271

However, it requires sub-queries, which Ecto < 2.0 doesn't support. So we've come up with a 2 queries solution, which looks like:

defp paginate_with_joins(q, config) do
    # First, find out the ids of the things we want instead
    stripped_query = q
                     |> exclude(:select)
                     |> exclude(:preload)

    offset = config.page_size * (config.page_number-1)
    page_size = config.page_size
    ids = stripped_query
          |> exclude(:group_by)
          |> select([x], {x.id})
          |> group_by([x], x.id)
          |> offset(^offset)
          |> limit(^page_size)
          |> Repo.all

    # Then find the actual results using the ids as selectors instead of
    # offset & limit
    as_uuids = Enum.map(ids, fn {str_id} -> str_id end)
    entries = q |> where([x], x.id in ^as_uuids) |> Repo.all

    # We also need the total count, so we find it out
    count = stripped_query
            |> exclude(:order_by)
            |> select([x], count(x.id, :distinct))
            |> Repo.one!

    # Then we create the Scrivener-compatible page
    %Scrivener.Page{
      page_size: config.page_size,
      page_number: config.page_number,
      entries: entries,
      total_entries: count,
      total_pages: ceiling(count, config.page_size)
    }
  end

Would you be interested in including some form of this in Scrivener? We could submit a PR.

Thanks,
Thomas

The Ecto 2.0.0-rc is out. Are there any plans to get scrivener working with the new version? I've noticed that ja_serializer depends on Scrivener also updating, (see vt-elixir/ja_serializer#112). Not sure what your plans are for the transition.

It would be practical to have the ability to load the last page whatever that page number might be.

Imagine that I have a ForumTopicController that loads a forum topic and all the replies to that topic:

forum_replies = assoc(conn.assigns[:forum_topic], :forum_replies)
                |> ForumReply.in_order
                |> preload(:user)
                |> Repo.paginate(page: sanitized_page(params["page"]))

Whenever I link to a forum topic from other pages, I would like to link to the last page (ie. the latest replies), not the first page. This could for instance be done by setting ?page=last.

Because of this code here

  def new(module, defaults, options) do
    options = normalize_options(options)
    page_number = options["page"] |> to_int(1)

    %Scrivener.Config{
      caller: Map.get(options, "caller", self()),
      module: module,
      options: merged_options(defaults, options),
      page_number: page_number,
      page_size: page_size(defaults, options)
    }
  end

When something like Repo.paginate(query, page_number: 2) is called it is always overridden by options["page"] |> to_int(1) to be 1. Given that :page_number is a key on the %Scrivener.Config{} struct and :page is not this is super confusing and not easily noticeable.

Compiling scrivener on Elixir 1.11 throws the following warning:

==> scrivener
Compiling 4 files (.ex)
warning: Scrivener.Paginater.Atom.__impl__/1 defined in application :scrivener_ecto is used by the current application but the current application does not directly depend on :scrivener_ecto. To fix this, you must do one of:

  1. If :scrivener_ecto is part of Erlang/Elixir, you must include it under :extra_applications inside "def application" in your mix.exs

  2. If :scrivener_ecto is a dependency, make sure it is listed under "def deps" in your mix.exs

  3. In case you don't want to add a requirement to :scrivener_ecto, you may optionally skip this warning by adding [xref: [exclude: Scrivener.Paginater.Atom]] to your "def project" in mix.exs

  lib/scrivener/paginater.ex:1: Scrivener.Paginater.impl_for/1

Generated scrivener app

I'm using scrivener_ecto and therefore have Scrivener.Config.new/3 be called in the Repo. Now I'd like to have some defaults set per controller as well and I'd like to simply use the functionality of Scrivener.Config instead of rolling my own:

def index(conn, params) do
  pagination_opts = Scrivener.Config.new(nil, [page_size: 2], params)
  
  entries = query |> Repo.paginate(pagination_opts)

  render conn, "index.html", entries: entries
end

This doesn't work, as Scrivener.Config doesn't implement Enumerable, which I feel like it could do without any major downsides.

Any chance we could get offset and limit support added?

Hi,

I am getting the error below

=> scrivener
Compiled lib/scrivener/page.ex

== Compilation error on file lib/scrivener.ex ==
** (CompileError) lib/scrivener.ex:152: cannot use ^offset outside of match clauses
    (elixir) expanding macro: Kernel.|>/2
    lib/scrivener.ex:155: Scrivener.entries/4
    (elixir) expanding macro: Kernel.if/2
    lib/scrivener.ex:147: Scrivener.entries/4

Here is my module

defmodule EventTrack.Event do
  use Ecto.Schema
  import Ecto.Query
  import Ecto.Changeset
  alias EventTrack.{Repo, Event}

  def search(params \\ :empty) do
    page = Event
      |> where([e], e.name == ^params["name"])
      |> order_by([e], desc: e.inserted_at)
      |> Repo.paginate(params)
    // omited
  end

deps

 {:ecto, "~> 1.0.7", override: true},
 {:mongodb_ecto, "~> 0.1.4"},
 {:scrivener, "~> 1.0"}

Thx