drewolson / scrivener Goto Github PK
View Code? Open in Web Editor NEWPagination for the Elixir ecosystem
License: MIT License
Pagination for the Elixir ecosystem
License: MIT License
I've detailed the problem at the Scrivener Elastic project [here] (radar/scrivener_elastic#1)
Basically I want to add a new field to the Config struct. If it's not set, it will return all the pages found( can be a large number ie 1_000_000). If it's set then it will show only the first max_number_of_pages pages.
Can we add this field it to Scrivener.Config?
I've also noticed this open issue that is a bit similar to my situation but it's for Scrivener Ecto
use case. Maybe we can find a way to avoid this types of issues in the future, since you need to change the Scrivener.Config
every time you need a new field for a package that just happens to use Scrivener.
Maybe a :options
/:custom_options
field that can hold custom parameters will do the trick. In this way we won't need to create extra fields in Scrivener for each use case and keep the package as slim and general as possible.
Hey I know in 2.20 :options
were introduced, but I was just wondering how I would go about actually using them? I want to limit the total number of pages returned for queries that can return millions of results. I've tried looking at the hexdocs but still can't figure it out.
One of the benefits of scrivener is being able to do Repo.paginate(query, params)
where params comes from a user input (in the case of Phoenix.) Currently it is possible for a user to set page_size
to a high value without restriction.
There are a few solutions I can think of outside of Scrivener:
Repo.paginate_with_limit
function which does the same thing and calls paginateNone of these options are quite as convenient as just passing through params
though. I was wondering if it is justified to have a max_page_size
config option? It could default to :infinity
which make the pagination behave as it does today.
Hey man! Hope things are good!
I'm running a local fork of Scrivener/Scrivener_Ecto right now that adds a show_totals: boolean
field to the Config struct.
The default is true, since I only turn it off for certain tables.
The use case is when working with unbounded data (like a sensor reading table in postgres, or gps points, etc), calculating the totals can cause DB timeouts.
In particular, I think pagination has really nice use cases w/ the new Flow module for processing large sets of data.
If you like the idea, I can submit a PR w/ tests to the repos monday! Let me know. Scrivener is solid by the way, thanks for creating it!
When trying to use the library I get
no function clause matching in Scrivener.HTML.pagination_links/2
my html looks like this
<div class="col-sm-12">
<h3 class="text-center">Recent Auctions</h3>
<%= for auction <- @recent_auctions do %>
<%= render "auction_block.html", auction: auction %>
<% end %>
<%= pagination_links @conn, @page %>
</div>
my controller looks like
query = from a in Auction,
order_by: [desc: a.time],
preload: [:state, :city, :facility]
page = Repo.paginate(query, params)
assign(conn, :user_city, nil)
|> assign(:auctions, page.entries)
|> assign(:page_number, page.page_number)
|> assign(:page_size, page.page_size)
|> assign(:total_pages, page.total_pages)
|> assign(:total_entries, page.total_entries)
|> assign(:recent_auctions, page.entries)
|> assign(:page, page.entries)
|> assign(:recent_auctions_for_user_city, [])
Any ideas what is going on?
The upgrade to new phoenix (1.1) fails with error:
Looking up alternatives for conflicting requirements on postgrex
Activated version: 0.9.1
From mix.exs: >= 0.0.0
From ecto v1.1.0: ~> 0.10
From scrivener v1.1.0: ~> 0.9.0
** (Mix) Hex dependency resolution failed, relax the version requirements or unlock dependencies
I have a case where the number of records is relatively large, so it produces e.g. 50+ pages. This cases a bit of an issue with UI since those pages don't fit the page nicely (break into multiple rows etc.).
For kaminari for example, this is configurable (even has default behavior) via various window
related params. So, is there a way to achieve the same for scrivener?
If not, would there be interest in adding this to scrivener or do you prefer to leave it to the consumers?
Thank you for your time ๐
P.S. If there'd be interest in adopting this for scrivener, I have to solve this for a project, might ask the owner to devote some time to add this to scrivener so I might help.
What's the best way to use this with a preloaded association?
Something like?
page = from c in Comment, order_by: c.published_at |> Repo.paginate(params)
Repo.all from p in Post, preload: [comments: ^page.entries]
Thanks for the code.
Was attempting to use to paginate a logs table I have that I'm ordering by timestamp but otherwise don't keep an ID. Perhaps consider allowing a configurable count field?
Thanks
Hi, I'm new to Elixir so trying to add small features to applications right now. Please excuse me if this is totally dumb, but I've spent a while trying to debug and I'm lost.
I started to work on rrrene/elixirstatus-web#13 which was to add pagination using Scrivener. Here is what I've done: rrrene/elixirstatus-web@master...philnash:pagination. I hoped that it would be mostly as simple as adding Scrivener and changing the call to Repo.all
to Repo.paginate
in the PostingsController
. However, I get the error Array type is not supported by MySQL
. The full stack trace is below:
[error] #PID<0.490.0> running ElixirStatus.Endpoint terminated
Server: localhost:4000 (http)
Request: GET /postings
** (exit) an exception was raised:
** (ArgumentError) Array type is not supported by MySQL
(ecto) lib/ecto/adapters/mysql/connection.ex:352: Ecto.Adapters.MySQL.Connection.expr/2
(elixir) lib/enum.ex:1008: anonymous fn/4 in Enum.map_join/3
(elixir) lib/enum.ex:1261: Enum."-reduce/3-lists^foldl/2-0-"/3
(elixir) lib/enum.ex:1007: Enum.map_join/3
(ecto) lib/ecto/adapters/mysql/connection.ex:347: Ecto.Adapters.MySQL.Connection.expr/2
(elixir) lib/enum.ex:1008: anonymous fn/4 in Enum.map_join/3
(elixir) lib/enum.ex:1261: Enum."-reduce/3-lists^foldl/2-0-"/3
(elixir) lib/enum.ex:1007: Enum.map_join/3
I can understand MySQL not being able to do some things, but this seems like a simple use case and I can't tell where or how I am trying to use the Array type.
Any help would be gladly received. Thanks.
Cannot pass a list of DB results to the Scrivener.paginate/2
function.
Sometimes I find that I get a list of results from the DB using Ecto and I need to do a some operation on the list outside the realm of Ecto and Ecto Queries.
With the finalized list post processing, I still need to paginate the list but currently scrivener does not support passing in a list for pagination.
Add ability to paginate a list.
I've started on this already and trying it out at the moment. I'd appreciate your feedback.
Thanks for writing this library BTW.
When in Add
use Scrivener, page_size: 10
to my repo I get the error
module Scrivener is not loaded could not be found, even after fetching the dependencies. I am using the latest phoenix for my project
The legal team at my company doesn't like us using repositories without license files in the repository itself. Would it be possible for you to add the MIT license file to the repo?
Since you already need to retrieve the count of entries to get the total number of pages, it would be beneficial to add it to the Page
struct so the user doesn't need to retrieve it again
Hi @drewolson, I'm having issues with dialyzer when using latest available hex package:
:0:unknown_function
Function Scrivener.paginate/2 does not exist.
________________________________________________________________________________
:0:unknown_function
Function Scrivener.Config.new/3 does not exist.
________________________________________________________________________________
:0:unknown_type
Unknown type: Scrivener.Page.t/0.
________________________________________________________________________________
done (warnings were emitted)
If I switch to using scrivener
and scrivener_ecto
from github master, then it works fine.
Could you please release a new version with these type fixes? ๐
Thanks! ๐
I had a similar problem than this guy.
I tried to make a query grouping by month, using fragment and date_trunc, it works with Repo.all() but with Repo.paginate() I get
expected at most one result but got 2 in query:
With this query:
query = from u in User,
join: r in assoc(u, :roles),
where: not r.name in ["a", "b"],
preload: [roles: r]
The sql returns 28 rows, but there's only 20 unique users.
%Scrivener.Page{entries: [...], page_number: 1, page_size: 10, total_entries: 28,
total_pages: 3}
When I iterate through the items page 1 has 10 items, page 2 has 7 items, and page 3 has 3 items.
I'm pretty new to Ecto, so maybe my query is wrong, but it looks like scrivener isn't recognizing the duplicate rows from the join.
Would be great to have an option to don't show the pagination when there is only 1 page with results.
My Application's Repo looks like this
defmodule AppName.Repo do
use Ecto.Repo, otp_app: :app_name
use Scrivener, page_size: 20
end
My index action looks like this...
def index(conn, params) do
words = from(w in Word, order_by: [asc: :abc_id], preload: [:languages])
|> Repo.paginate(params)
render(conn, "index.json",
%{
words: words.entries,
current_page: words.page_number,
total_pages: words.total_pages,
per_page: words.page_size
}
)
end
Versions of other notable deps in mix file.
scrivener 1.0.0
phoenix 1.0.2
phoenix_ecto 1.2.0
ecto 1.0.3
P.S: Scrivener.paginate/2 works with %Scrivener.Config.
After upgrading from version 2.7.0 to version 2.7.1 we have the following compilation error:
Dependencies have diverged:
* ex_doc (Hex package)
the :only option for dependency ex_doc
> In mix.exs:
{:ex_doc, "~> 0.22", [env: :prod, repo: "hexpm", hex: "ex_doc", only: :dev, runtime: false]}
does not match the :only option calculated for
> In deps/scrivener/mix.exs:
{:ex_doc, ">= 0.0.0", [env: :prod, hex: "ex_doc", repo: "hexpm", optional: false]}
Remove the :only restriction from your dep
** (Mix) Can't continue due to errors on dependencies
It seems it didn't like this change: https://diff.hex.pm/diff/scrivener/2.7.0..2.7.1#84732810-30-
Thank you for your time!
Hello, I had pagination working fine in my app using:
But now, after upgrading to:
It's failing with:
[error] #PID<0.4760.0> running MyApp.Endpoint terminated
Server: localhost:4000 (http)
Request: GET /s
** (exit) an exception was raised:
** (Protocol.UndefinedError) protocol Scrivener.Paginater not implemented for #Ecto.Query<from s in MyApp.Sitemap, where: s.user_id == ^1, order_by: [desc: s.inserted_at]>
lib/scrivener/paginater.ex:1: Scrivener.Paginater.impl_for!/1
lib/scrivener/paginater.ex:10: Scrivener.Paginater.paginate/2
(MyApp) web/controllers/sitemap_controller.ex:28: MyApp.SitemapController.index/2
(MyApp) web/controllers/sitemap_controller.ex:1: MyApp.SitemapController.action/2
(MyApp) web/controllers/sitemap_controller.ex:1: MyApp.SitemapController.phoenix_controller_pipeline/2
(MyApp) lib/MyApp/endpoint.ex:1: MyApp.Endpoint.instrument/4
(MyApp) lib/phoenix/router.ex:261: MyApp.Router.dispatch/2
(MyApp) web/router.ex:1: MyApp.Router.do_call/2
(MyApp) lib/MyApp/endpoint.ex:1: MyApp.Endpoint.phoenix_pipeline/1
(MyApp) lib/plug/debugger.ex:93: MyApp.Endpoint."call (overridable 3)"/2
(MyApp) lib/MyApp/endpoint.ex:1: MyApp.Endpoint.call/2
(plug) lib/plug/adapters/cowboy/handler.ex:15: Plug.Adapters.Cowboy.Handler.upgrade/4
(cowboy) src/cowboy_protocol.erl:442: :cowboy_protocol.execute/4
I've already cleaned up and reinstalled the deps, not sure what is failing here. This is the code in the controller:
def index(conn, params) do
sitemaps = Sitemap
|> where(user_id: ^(current_user(conn).id))
|> order_by(desc: :inserted_at)
|> Repo.paginate(params)
render(conn, "index.html", sitemaps: sitemaps)
end
Any idea?
Thanks!
(Comes from mgwidmann/scrivener_html#30)
The following statement works with 1.1.1
iex(1)> (from c in Comment, join: u in assoc(c, :user), order_by: [desc: u.inserted_at]) |> Repo.paginate(page: 1, page_size: 10)
[22:43:12.618] [debug] SELECT count('*') FROM "comments" AS c0 INNER JOIN "users" AS u1 ON u1."id" = c0."user_id" [] OK query=41.0ms
[22:43:12.679] [debug] SELECT c0."id", c0."content", c0."username", c0."email", c0."deleted", c0."approved", c0."inserted_at", c0."updated_at", c0."user_id", c0."post_id", c0."parent_id" FROM "comments" AS c0 INNER JOIN "users" AS u1 ON u1."id" = c0."user_id" ORDER BY u1."inserted_at" DESC LIMIT $1 OFFSET $2 [10, 0] OK query=60.7ms queue=0.1ms
but raises an error with 1.1.3
iex(4)> (from c in Comment, join: u in assoc(c, :user), order_by: [desc: u.inserted_at]) |> Repo.paginate(page: 1, page_size: 10)
[22:55:14.153] [debug] SELECT count(DISTINCT c0."id") FROM "comments" AS c0 INNER JOIN "users" AS u1 ON u1."id" = c0."user_id" [] OK query=69.8ms
[22:55:14.160] [debug] SELECT c0."id" FROM "comments" AS c0 INNER JOIN "users" AS u1 ON u1."id" = c0."user_id" GROUP BY c0."id" ORDER BY u1."inserted_at" DESC LIMIT $1 OFFSET $2 [10, 0] ERROR query=6.8ms queue=0.1ms
** (Postgrex.Error) ERROR (grouping_error): column "u1.inserted_at" must appear in the GROUP BY clause or be used in an aggregate function
(ecto) lib/ecto/adapters/sql.ex:185: Ecto.Adapters.SQL.query!/5
(ecto) lib/ecto/adapters/sql.ex:481: Ecto.Adapters.SQL.execute/6
(ecto) lib/ecto/repo/queryable.ex:95: Ecto.Repo.Queryable.execute/5
(ecto) lib/ecto/repo/queryable.ex:15: Ecto.Repo.Queryable.all/4
(scrivener) lib/scrivener.ex:153: Scrivener.entries/4
(scrivener) lib/scrivener.ex:102: Scrivener.paginate/2
Is there any plan to disable "select count(*)" by param?,
I use automatic pagination on my website, like facebook (when you research end of page, it load another one) and I'd like to not did 2 selects all the time.
my logic is:
get next page with 30 items
So, my client will make another ajax call only when last call returned exactly 30 items. It's a performace gain!
tip: I look you code, and you are calling total_entries before entries. You could test if entries returned less than page_size and just calculate total_entries. Will be 1 select less on last page ;)
Please let me know if this isn't the appropriate venue โ or just close the issue :) โ I also noticed that this project itself is in 'low maintenance mode', so no worries if this isn't something you'd be interested in in helping out with.
Scrivener.HTML
seems to be un-maintained; see:
Several users of that library have contributed changes to resolve some issues, mainly due to dependency conflicts as various other libraries have been updated. I think most of the desired changes are literally just fiddling with dependency versions, e.g. relaxing version requirements.
Unfortunately, the owner of that project/package doesn't seem able or willing to follow-up on any of the open issues. (And that's totally okay!) But perhaps you, the creator/owner/maintainer and various contributors to this project, might be able to contact the creator/owner of Scrivener.HTML
yourself and ask them to consider transferring ownership of the project to someone else. (And it's totally fine if no one has the time, or wants to spend it, to do any of that.)
Worst case, the users of this library can always fork it themselves โ several have done so already.
Maybe second-best case would be someone stepping up to maintain a fork. That seems like it would also require changing the project/package name (e.g. on Hex).
Thanks for the project!
Having issues with one of my functions because I try to do .entries
on a Scrivener map. Any ideas what the below means? Any help at all is absolutely appreciated.
#{'entries' := __@1} can never match the type
A little bit more info about what I'm doing is here. https://elixirforum.com/t/handle-adding-fields-into-map/31846
Hey,
We've stumbled upon an issue with Scrivener and joined models counting, which prevents correct pagination.
Here is how it works:
Article
is our model with many relationships)result = Article |> Article.with_tags_and_assets |> Repo.paginate(%{"page_size"=> 3})
pry(26)> result.entries |> Enum.count
1
# BUT
result = Article |> Article.with_tags_and_assets |> Repo.paginate(%{"page_size"=> 300})
pry(28)> result.entries |> Enum.count
3
# with_tags_and_assets is:
def with_tags_and_assets(q) do
q
|> join(:left, [a], t in assoc(a, :tags))
|> join(:left, [a], i in assoc(a, :images))
|> join(:left, [a], d in assoc(a, :documents))
|> join(:left, [a], v in assoc(a, :videos))
|> preload([a, t, i, d, v], [tags: t, images: i, documents: d, videos: v])
end
# And the relationships definitions in Article
has_many :article_tags, MyApp.Models.ArticleTag
has_many :tags, through: [:article_tags, :tag]
has_many :article_assets, MyApp.Models.ArticleAsset
has_many :images, through: [:article_assets, :image]
has_many :documents, through: [:article_assets, :document]
has_many :videos, through: [:article_assets, :video]
This is caused by a mismatch between the number of rows returned by the DB, and the actual number of entities that you get, since with joins, 1 entity >= 1 row.
Especially:
There is a SQL solution, presented here: http://stackoverflow.com/questions/15897055/proper-pagination-in-a-join-select/15897271#15897271
However, it requires sub-queries, which Ecto < 2.0 doesn't support. So we've come up with a 2 queries solution, which looks like:
defp paginate_with_joins(q, config) do
# First, find out the ids of the things we want instead
stripped_query = q
|> exclude(:select)
|> exclude(:preload)
offset = config.page_size * (config.page_number-1)
page_size = config.page_size
ids = stripped_query
|> exclude(:group_by)
|> select([x], {x.id})
|> group_by([x], x.id)
|> offset(^offset)
|> limit(^page_size)
|> Repo.all
# Then find the actual results using the ids as selectors instead of
# offset & limit
as_uuids = Enum.map(ids, fn {str_id} -> str_id end)
entries = q |> where([x], x.id in ^as_uuids) |> Repo.all
# We also need the total count, so we find it out
count = stripped_query
|> exclude(:order_by)
|> select([x], count(x.id, :distinct))
|> Repo.one!
# Then we create the Scrivener-compatible page
%Scrivener.Page{
page_size: config.page_size,
page_number: config.page_number,
entries: entries,
total_entries: count,
total_pages: ceiling(count, config.page_size)
}
end
Would you be interested in including some form of this in Scrivener? We could submit a PR.
Thanks,
Thomas
The Ecto 2.0.0-rc is out. Are there any plans to get scrivener working with the new version? I've noticed that ja_serializer depends on Scrivener also updating, (see vt-elixir/ja_serializer#112). Not sure what your plans are for the transition.
It would be practical to have the ability to load the last page whatever that page number might be.
Imagine that I have a ForumTopicController
that loads a forum topic and all the replies to that topic:
forum_replies = assoc(conn.assigns[:forum_topic], :forum_replies)
|> ForumReply.in_order
|> preload(:user)
|> Repo.paginate(page: sanitized_page(params["page"]))
Whenever I link to a forum topic from other pages, I would like to link to the last page (ie. the latest replies), not the first page. This could for instance be done by setting ?page=last
.
Because of this code here
def new(module, defaults, options) do
options = normalize_options(options)
page_number = options["page"] |> to_int(1)
%Scrivener.Config{
caller: Map.get(options, "caller", self()),
module: module,
options: merged_options(defaults, options),
page_number: page_number,
page_size: page_size(defaults, options)
}
end
When something like Repo.paginate(query, page_number: 2)
is called it is always overridden by options["page"] |> to_int(1)
to be 1
. Given that :page_number
is a key on the %Scrivener.Config{}
struct and :page
is not this is super confusing and not easily noticeable.
Compiling scrivener on Elixir 1.11 throws the following warning:
==> scrivener
Compiling 4 files (.ex)
warning: Scrivener.Paginater.Atom.__impl__/1 defined in application :scrivener_ecto is used by the current application but the current application does not directly depend on :scrivener_ecto. To fix this, you must do one of:
1. If :scrivener_ecto is part of Erlang/Elixir, you must include it under :extra_applications inside "def application" in your mix.exs
2. If :scrivener_ecto is a dependency, make sure it is listed under "def deps" in your mix.exs
3. In case you don't want to add a requirement to :scrivener_ecto, you may optionally skip this warning by adding [xref: [exclude: Scrivener.Paginater.Atom]] to your "def project" in mix.exs
lib/scrivener/paginater.ex:1: Scrivener.Paginater.impl_for/1
Generated scrivener app
I'm using scrivener_ecto and therefore have Scrivener.Config.new/3
be called in the Repo. Now I'd like to have some defaults set per controller as well and I'd like to simply use the functionality of Scrivener.Config
instead of rolling my own:
def index(conn, params) do
pagination_opts = Scrivener.Config.new(nil, [page_size: 2], params)
entries = query |> Repo.paginate(pagination_opts)
render conn, "index.html", entries: entries
end
This doesn't work, as Scrivener.Config
doesn't implement Enumerable
, which I feel like it could do without any major downsides.
Any chance we could get offset and limit support added?
Hi,
I am getting the error below
=> scrivener
Compiled lib/scrivener/page.ex
== Compilation error on file lib/scrivener.ex ==
** (CompileError) lib/scrivener.ex:152: cannot use ^offset outside of match clauses
(elixir) expanding macro: Kernel.|>/2
lib/scrivener.ex:155: Scrivener.entries/4
(elixir) expanding macro: Kernel.if/2
lib/scrivener.ex:147: Scrivener.entries/4
Here is my module
defmodule EventTrack.Event do
use Ecto.Schema
import Ecto.Query
import Ecto.Changeset
alias EventTrack.{Repo, Event}
def search(params \\ :empty) do
page = Event
|> where([e], e.name == ^params["name"])
|> order_by([e], desc: e.inserted_at)
|> Repo.paginate(params)
// omited
end
deps
{:ecto, "~> 1.0.7", override: true},
{:mongodb_ecto, "~> 0.1.4"},
{:scrivener, "~> 1.0"}
Thx
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.