sheharyarn / memento Goto Github PK

I’ve created a module of CRUD operations, mostly one-liners, but I’m stuck at creating a general purpose update function.
Goals:
1. Update the value of any key or keys in the row, independently of each other.
2. Insert and or update the :updated_at key.
3. Only update the keys specified by the update functions arguments

My approach so far has been something along these lines (pseudo code):

defmodule Todo do
  use Memento.Table,
    attributes: [:id, :name, :priority, :updated_at],
    type: :ordered_set,
    autoincrement: true

# .....

  def update(id, changes = %{}) do
    Memento.transaction do
      case get_by_id(id) do
        %Todo{} = todo ->
          todo
          |> adjust( changes)
          |> Map.put!(:updated_at, NaiveDateTime.utc_now()
 
        _ ->
          {:error, :not_found}
      end
    end
    
    defp adjust(%Todo{} = todo, changes) do
       todo
         |> Map.update!(changes))
         |> Todo.write()
    end

    def get_by_id(id) do
          Memento.transaction(fn -> Memento.Query.read(Todo, id) end)
    end

Elixir school demonstrates updates with Mnesia.write/1 but this overwrites the whole row.

The other solutions I found are either over my head or in Erlang:

• https://stackoverflow.com/questions/63048415/elixir-mnesia-what-is-the-most-concise-way-to-update-a-set-of-value-in-an-el
• https://stackoverflow.com/questions/1820996/erlang-mnesia-updating-a-single-field-value-in-a-row

Hello

Im trying to build a release of an application that uses Memento.
When running the app with mix phx.server everything works as expected.
When I try to create a production release, I get an error when trying to access the store.
%Memento.Error{message: "Transaction Failed with: {:no_exists, Sigfox.DeviceState}"}

Any help would be much appreciated.

Thanks

Didn't find anywhere if memento supports composite primary keys, mnesia does support it would it be possible to bring that into memento ?

I tried to create Distillery release commands, but gave up because the node is not set. Even if I replace the database path or move the files, the database is not usable when the application is started with the proper node name.

There may or may not have been issues with mix commands as well. I did not test them as extensively, but I am using the same base function for both.

Is there a way to create a usable Mnesia database for the node real_node_name@real_host_name when the current node is nonode@nohost in a Distillery release task? Also, being able to batch create Mnesia databases for a list of nodes might be useful, although I do not think that changes anything given a solution for one node.

EDIT

I wound up using the following function to set set the node name for the task.

def set_node_name_for_task(node_name \\ nil) do
  node_name
  |> Kernel.||(System.get_env("NODE_NAME"))
  |> Kernel.||("nonode@nohost")
  |> String.to_atom
  |> Node.start
end

Combined with the following change to the :mnesia configuration, release tasks can be used to create databases. This only works for a single node at a time, however.

config :mnesia,
  dir: '/etc/.mnesia/#{Mix.env}/#{System.get_env("NODE_NAME")}'        # Notice the single quotes

Maybe it is worth adding a note to the documentation?

Maybe I did get something wrong, but how do I persist data on disk across restarts? I reread the documentation several times, tried to reuse :mnesia.create_table options... And nothing - data exists only in RAM or error is raised.

Due to the shutdown of travis-ci.org, all existing pipelines will be dead soon, so it makes sense to switch to Github Actions, which will remain free for OS projects.

Calling Memento.Query.select_raw(...) leads to an error when attempting to load the results.

** (FunctionClauseError) no function clause matching in Memento.Query.Data.load/1    
    
    The following arguments were given to Memento.Query.Data.load/1:
    
        # 1
        "a"
    
    Attempted function clauses (showing 1 out of 1):
    
        def load(data) when is_tuple(data)
    
    (memento 0.3.2) lib/memento/query/data.ex:55: Memento.Query.Data.load/1
    (elixir 1.13.0) lib/enum.ex:1593: Enum."-map/2-lists^map/1-0-"/2
    (mnesia 4.20.3) mnesia_tm.erl:842: :mnesia_tm.apply_fun/3
    (mnesia 4.20.3) mnesia_tm.erl:818: :mnesia_tm.execute_transaction/5
    (memento 0.3.2) lib/memento/transaction.ex:71: Memento.Transaction.execute/2

Add support for sigils and module attributes by expanding the arguments given to the Memento macros. See this stackoverflow question.

Question sent to me via email:

How would I guarantee that Memento/Mnesia is replicating data across nodes, I’m using something called peerage which does node discovery, so at boot the nodes may not all be discovered.

A question was posted on the ElixirForum today:

I’m using @sheharyarn’s Memento for integrating Mnesia for my Phoenix App. Having little to no experience in erlang, where should I look in the documentation for complex queries, I am storing a map in one of my column and I need to fetch by querying in that map, I can do this in Ecto using fragments but I’m wondering how to do the same using Mnesia.

The Erlang Matchspec is very confusing, especially so for beginners. It becomes even harder to write them when nested maps are involved. Quite often, I also forget how to use them for many scenarios and have to read the Erlang docs on match_spec again.

This was one of the very reasons I decided to write the Memento library in the first place. While I won't add support for an API to directly query nested maps in the package as of yet, I'll use this issue to cover some of the advanced queries for now.

@sheharyarn this is a real cool effort you've got here. 👏 Nice work.

How do I link up two nodes together using Memento? I don't see any change config or add table copy logic so far. Am I correct that I should drop out to :mnesia to do it?

Eg (w/two node RAM instances)... first node:

Memento.start
Memento.Table.create!(MyLovelyTable)
#:mnesia.wait_for_tables?

Second node:

Memento.start
:mnesia.change_config(:extra_db_nodes, Node.list())
:mnesia.add_table_copy(MyLovelyTable, node())
#:mnesia.wait_for_tables?

Hi, I really like this library and have been using it to build a prototype application. I am wondering if there is a way to paginate?

I am using memento to wrap Mnesia functionality into my application.

I want to check if a record with a certain id exists and if not perform certain actions.

Documentation says read should return nil when record doesn’t exist. Howerver, the Memento.transation wrapper aborts with a:not_existsexception therefore my application cannot proceed.

Here’s how my function looks like:

def find_conversation(user_id) do
    Memento.transaction! fn ->
      user_id
      |> read
      |> to_conversation
    end
end

defp to_conversation(nil), do: nil
  defp to_conversation(%@store{} = conversation) do
    struct(Conversation, Map.from_struct(conversation))
  end

Hey @sheharyarn

I understand migrations are not fully supported yet, is there a way to make it work? Let's say I add a field i want it to have the new fields able to be stored. Any procedure / steps you recommend to manually do migrations?

Hi there,
I am using Memento for a small project. It works great but I am lacking a small thing. Is there a way to search for a pattern ?

For example I have list of Movies containing word "Rush". So in a traditional database, we will do something like

where title like '%Rush%'

Is there something similar with Mnesia?

Regards,

I'm using que without persistence. Currently after upgrading Que from 0.5.0 to 0.7.0 which changed dependency from mnesia to memento, I'm getting these errors:

Elixir.Memento.MnesiaException: Mnesia operation failed
   {:cyclic, :"[email protected]", {Que.Persistence.Mnesia.DB.Jobs, :______WHOLETABLE_____}, :read, :read, {:tid, 138, #PID<0.5537.0>}}
   Mnesia Error: {:cyclic, :"[email protected]", {Que.Persistence.Mnesia.DB.Jobs, :______WHOLETABLE_____}, :read, :read, {:tid, 138, #PID<0.5537.0>}}
  File "lib/memento/query/query.ex", line 728, in Memento.Query.autoincrement_key_for/1
  File "lib/memento/query/query.ex", line 701, in Memento.Query.prepare_record_for_write!/2
  File "lib/memento/query/query.ex", line 250, in Memento.Query.write/2
  File "lib/que/persistence/mnesia/db.ex", line 144, in anonymous fn/1 in Que.Persistence.Mnesia.DB.Jobs.update_job/1
  File "mnesia_tm.erl", line 836, in :mnesia_tm.apply_fun/3
  File "mnesia_tm.erl", line 812, in :mnesia_tm.execute_transaction/5
  File "lib/memento/transaction.ex", line 71, in Memento.Transaction.execute/2
  File "lib/memento/transaction.ex", line 84, in Memento.Transaction.execute!/2
  Module "Elixir.Que.Server", in Que.Server.init/1

Seems like there's a cyclic error in the transaction

Related to #15 and #17.

Add more mnesia wrapper methods to easily add new cluster nodes to the mnesia config and change schema, copy type, waiting for tables to get ready and more.

• Connect Nodes
  • Memento.add_nodes(nodes)
• Schema wait
• Schema copy
  • Memento.Schema.set_storage_type(node, mode)
• Table copy
  • Memento.Table.create_copy(Jobs, node, mode)
  • Memento.Table.delete_copy(Jobs, node)
  • Memento.Table.move_copy(Jobs, from_node, to_node)
  • Memento.Table.set_storage_type(Jobs, node, mode)

I noticed that if a query fails to return any records, I get the following error:
** (FunctionClauseError) no function clause matching in Memento.Query.coerce_records/1
(memento) lib/memento/query/query.ex:663: Memento.Query.coerce_records(:"$end_of_table")

However calling Mnesia directly, I get the standard message of:
{:atomic, []} that I can easily pattern match on.

Hi! First of all, thanks for this library, is awesome!
I have a little problem querying Mnesia by date: I need to get every record where :creation_date is less than 24 hours ago.

I am writing the following method:

    def select_by_date do
     date = Timex.shift(NaiveDateTime.utc_now, hours: -24) 
     run_query(
        {:<, :created_at, date}
      )
    end

    defp run_query(pattern) do
      Memento.transaction! fn ->
        @store 
        |> Memento.Query.select(pattern)
      end
    end

For the run_querymethod, I've followed your example.

When I run the select_by_date method, I get the following error:

iex(1)> delete_outdated_config
** (Memento.MnesiaException) Mnesia operation failed
   Bad or invalid argument, possibly bad type
   Mnesia Error: {:badarg, [Engine.Mnesia.Db.Config, [{{Engine.Mnesia.Db.Config, :"$1", :"$2", :"$3", :"$4", :"$5", :"$6", :"$7"}, [{:<, :"$6", ~N[2019-01-23 16:50:45.690805]}], [:"$_"]}]]}
    (memento) lib/memento/query/query.ex:580: Memento.Query.select_raw/3
    (mnesia) mnesia_tm.erl:836: :mnesia_tm.apply_fun/3
    (mnesia) mnesia_tm.erl:812: :mnesia_tm.execute_transaction/5
    (memento) lib/memento/transaction.ex:71: Memento.Transaction.execute/2
    (memento) lib/memento/transaction.ex:84: Memento.Transaction.execute!/2

Is there a way to compare dates during the query?

I have few issues rolling out Memento on my application.

Implementation

1. Application has 3+ pods at any point of time and is connected with Libcluster.
2. In application.ex children are listed as follows
   a. {Cluster.Supervisor, [get_topologies(), [name: MyApp.ClusterSupervisor]]}
   b. MyApp.Mnesia.MigrateTask.child_spec()

defmodule MyApp.Mnesia.MigrateTask do
  def child_spec() do
    %{
      id: MODULE,
      start: {Task, :start_link, [fn -> setup() end]},
      type: :worker,
      restart: :temporary
    }
  end
  
  def setup() do
    # I inspected here, all the nodes were connected here.  
    nodes = [node() | Node.list()]
    
    # Stop Memento all nodes
    :rpc.multicall(nodes, Memento, :stop, [])
    
    # Create schema on all nodes
    Memento.Schema.create(nodes)
    
    # Start Memento all nodes
    :rpc.multicall(nodes, Memento, :start, [])
    
    # Create ram copies on all nodes
    Memento.Table.create!(MyApp.MetaSchema, ram_copies: nodes)        
  end
end  

Table Schema

defmodule QueryQuest.Mnesia.MetaSchema do
  use Memento.Table, attributes: [:key, :value]
end

3. After the pods are up, through remote console checked Mememto.info. I see all pods are on running db nodes.

Issue

But I queryall for the Table.

    Memento.transaction!(fn ->
      Memento.Query.all(MyApp.MetaSchema)
    end)

Error

** (Memento.Error) Transaction Failed with: {:no_exists, MyApp.MetaSchema}
    (memento 0.3.2) lib/memento/transaction.ex:178: Memento.Transaction.handle_result/1

I'm trying to debug it, but its not succesful.

Are there any plans to support table fragmentation?

Original post by @cpilka, moved from #3:

I have a weird issue with disc persistence. Data is flushed to disc just for a certain amount of records (few hundred). Single inserts are not written to disc. Just wonder if there's a :disc_only_copies in Memento that would write all data straight to mnesia files.

This one doesn't write to disc:

Memento.transaction! fn ->
  Memento.Query.write(record)
end

This one does:

Memento.transaction! fn ->
  for _ <- 1..100 do
    Memento.Query.write(record)
  end
end

Is there any setting that tells Mnesia to flush the memory part always to disc and not to flush reaching certain buffer? Or whatever causes this issue ...

Hello,
According to mnesia documentation. Mnesia has the ability to support, ram, disc_only as well as disc_copies.

ram_copies

This option makes it so all data is stored exclusively in ETS, so memory only. Memory should be limited to a theoretical 4GB (and practically around 3GB) for virtual machines compiled on 32 bits, but this limit is pushed further away on 64 bits virtual machines, assuming there is more than 4GB of memory available.

disc_only_copies

This option means that the data is stored only in DETS. Disc only, and as such the storage is limited to DETS' 2GB limit.

disc_copies

This option means that the data is stored both in ETS and on disk, so both memory and the hard disk. disc_copies tables are not limited by DETS limits, as Mnesia uses a complex system of transaction logs and checkpoints that allow to create a disk-based backup of the table in memory.

Learn You Some Erlang for Mnesia
I was wondering if Memento had the same support. I'm referencing this section of the mnesia documentation. Thank you!