Coder Social home page Coder Social logo

contacts.cert.at's People

Contributors

aaronkaplan avatar

Stargazers

avatar

Watchers

avatar avatar

Forkers

certat

contacts.cert.at's Issues

Use of sql transaction to prevent empty tables

In this script for example:

contacts.cert.at/at-cidr-ranges-by-maxmind/load.sh

Lines 8 to 19 in d6c16ee

if [ ${delete_data} == 1 ]; then
psql $DB -c "delete from networks;"
else
echo "warning: might have duplicate data (depending on PK uniqueness) since we are not deleting the previous import."
fi
# quirks needed for postgresql. Needs to be readable for postgresql user
mv ./networksv4.csv $path
mv ./networksv6.csv $path
psql $DB -c "COPY networks FROM '${path}/networksv4.csv' WITH CSV HEADER;"
psql $DB -c "COPY networks FROM '${path}/networksv6.csv' WITH CSV HEADER;"

The delete operation happens at the beginning, then there are other operations and two separate "inserts". So there is
a) always a small time frame where the database is empty and queries on it will falsely give no results. Think of stream processing: then there are very regular queries and lookups can then contain wrong data resulting in wrong decisions
b) In case of errors between, the table will be empty at all

As far as I see, all three statements (in this example) can be executed in just one psql-call, then you have the advantage of transactions and all changes are applied atomically.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.