davedash / django-fixture-magic Goto Github PK

View Code? Open in Web Editor NEW

385.0 385.0 98.0 93 KB

Utilities to extract and manipulate Django Fixtures.

License: BSD 3-Clause "New" or "Revised" License

Python 99.58% Makefile 0.42%

django-fixture-magic's People

Contributors

Stargazers

Watchers

Forkers

carljm fourk botondus yeago chrisspen yifanzhang danxshap i-trofimtschuk leetrout pratyushmittal kippr orientalperil callowayproject troyharvey ghostrocket leadsift mgalgs myszmen charleschen jackiekazil ercanbasa amplify-analytics athom09 momirza offlinelabs mehranakk kaitlin suprotkin caffodian shipci loppear belphanior dmicanzerofox doordash fallenhitokiri jonthdiaz cgdeboer allay creekfish punchagan kevingrahamfoster dwatkinsweb duduklein adrienemery vineelyalamarthy paytm dineshs91 sknutsonsf chenliangomc cb109 miigotu marciusvinicius jason-utilityapi kd2718 antman261 maciej-jaworski solarmonkey nektor211 tony jter farialima maxbo taleinat arslanhashmi chr15m andyzickler makingspace beatboxchad jdnrg jm2242 hancush henrybaxter krectra shivavogo cromat chalomobility geckon tylkocom sobolevn jimmy927 ezri-mudde insspb 0therguys bryanlandia kozikowskik amir-hadzic spry-group iq-scm xrmx aopas

django-fixture-magic's Issues

unexpected keyword argument 'use_natural_foreign_keys'

I added its entry(django-fixtur-magic) in "MY" requirements.txt file and also added 'fixture_magic' to the list of INSTALLED_APPS in MY application's settings.py.

Then I ran the command
./manage.py dump_object hmi.odeskmenutask 1901783 > test_fixture.json.

1901783 is its primary key.
This causes an error like this:
TypeError: init() got an unexpected keyword argument 'use_natural_foreign_keys'.

dump_object with no arguments throws an ugly error.

When you enter ./manage.py dump_object without further args, it just errors out.

Is this project still being developed?

I'm currently making some changes to meet my requirements. I have at present hard coded things, but was thinking about building it up to be configurable/have tests etc, but if the project is no longer being developed maintained, it wouldn't really be worth it. So is there an intent to keep developing?

dump_objects raises a FieldError when the primary key is not "id"

dump_object assumes the pk is named "id" which is not necessarily true. (How I wish it was! But sadly...)

Please update PyPI

Please bump the version number and make a new release to PyPI so the newest changes are retrievable.

Also README.rst says:

./manage.py dump_object APP.MODEL * > my_new_fixture.json

This should be written with quotes around the asterisk:

./manage.py dump_object APP.MODEL '*' > my_new_fixture.json

dump_object does not follow object relations

Environment: Django 3.0.6

In my case the app does not follow relations when exporting data with dump_object

Simplified models:

class Member(BaseModel):

    email = models.EmailField(unique=True)

class Subscription(BaseModel):

    member = models.ForeignKey(Member, on_delete=models.CASCADE)

When I execute python manage.py dump_object members.member --query '{"pk__in": [1]} for a Member with several Subscriptions I only get the Member model dumped:

[
{
    "model": "accounts.member",
    "pk": 1,
    "fields": {
        "email": "[email protected]",
    }
}
]

New django adds a label arg, changes arguments to fixture magic

In zamboni we recently upgraded django:

 $ git submodule status src/django
 40e28df9e19bcf80525010df44fea9eeffec54b4 src/django (1.2.1-1774-g40e28df)

After that fixture magic did not work with this command:

./manage.py dump_object --kitchensink zadmin.ValidationResult 793

It displayed an error about trying to load 793 as a model and finally I realized that app label was getting passed in as an arg by default. This command got it to work:

./manage.py dump_object --kitchensink zadmin zadmin.ValidationResult 793

(the app label had to be doubled)

Could not dump objects until I modified dump_object.py as per comment

Everything I tried would yield the following error, even on internal Django models:

CommandError: object_class must be provided in the following format: app_name.model_name
Try calling dump_object with --help argument or use the following arguments:
 <[--kitchensink | -k] [--natural] [--query] object_class id [id ...]>

Then I tried the fix from this comment on closed issue #45 and after that I was able to dump objects successfully.

I don't understand what that args line is doing. I could submit a PR removing it but it would be good to know why it is there first.

Django version is 1.9.1.

implement CI workflow to publish to pypi.

merge_fixtures is having some problem parsing parameters

python:2.7.5
version:0.1.3
errror:

usage: manage.py merge_fixtures [-h] [--version] [-v {0,1,2,3}]
                                [--settings SETTINGS]
                                [--pythonpath PYTHONPATH] [--traceback]
                                [--no-color]
manage.py merge_fixtures: error: unrecognized arguments:

Issue while dumping a single object

I was trying to get a sample fixture for an object and got the following:

python manage.py dump_object "collector.TwitterData" 492784

/usr/local/lib/python2.7/dist-packages/fixture_magic/management/commands/dump_object.py:9: RemovedInDjango19Warning: The utilities in django.db.models.loading are deprecated in favor of the new application loading system.
from django.db.models import loading

CommandError: object_class must be provided in the following format: app_name.model_name
Try calling dump_object with --help argument or use the following arguments:
<[--kitchensink | -k] [--natural] [--query] object_class id [id ...]>

I'm using django 1.8 for this project

Pip installs wrong version from PyPi

Pip does find 0.1.4:

$ pip search django-fixture-magic | grep django-fixture-magic
django-fixture-magic (0.1.4)                - A few extra management tools to handle fixtures.

But when I try to install pinned:

$ pip install django-fixture-magic==0.1.4
Collecting django-fixture-magic==0.1.4
  Could not find a version that satisfies the requirement django-fixture-magic==0.1.4 (from versions: 0.0.2.macosx-10.6-i386, 0.1.4.macosx-10.13-x86_64, 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.0.7, 0.0.8, 0.1.0, 0.1.1, 0.1.2, 0.1.3)
No matching distribution found for django-fixture-magic==0.1.4

And unpinned:

$ pip install django-fixture-magic
Collecting django-fixture-magic
Installing collected packages: django-fixture-magic
Successfully installed django-fixture-magic-0.1.3

I don't know why this happens, but it happens in any environment that I've tried. Can you fix this? Is there anything that I can do to help?

setup.py should exclude the tests package.

For example:

packages=find_packages(exclude=['tests']),

Reference: https://pythonhosted.org/setuptools/setuptools.html#using-find-packages

Thanks.

reorder_json does not ensure order

The reorder_json is very useful but python dict are not ordered by default or at least in python 2.7 (cf.: https://github.com/davedash/django-fixture-magic/blob/master/fixture_magic/utils.py#L7).

ipdb> bucket
{'tagging.tag': [], 'account.profile': [], 'account.user': [], 'document.document': [], 'sharing.sharing': []}
ipdb> models
['account.user', 'account.profile', 'tagging.tag', 'document.document', 'sharing.sharing']

May be I'm wrong but at this state the method is useless, or works sometimes. We should may be use OrderedDict for that: https://docs.python.org/2/library/collections.html#collections.OrderedDict

dump_object does not work with OneToOne fields

For an example, try dumping an add-on that's a persona on AMO with --kitchensink=yes .

Update PyPi

Howdy,
Looks like it has been a while since PyPi has been updated (May 2013). Can you update with the latest version so I can get access to the --query argument? Thanks!

dump_object fails on '*'

The dump_object command fails when receiving '*', succeeds when receiving individual IDs.

$ ./manage dump_object service.model '*'
/service/venv/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Traceback (most recent call last):
  File "django/manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/service/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/service/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/service/venv/lib/python3.6/site-packages/django/core/management/base.py", line 316, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/service/venv/lib/python3.6/site-packages/django/core/management/base.py", line 353, in execute
    output = self.handle(*args, **options)
  File "/service/venv/lib/python3.6/site-packages/fixture_magic/management/commands/dump_object.py", line 117, in handle
    add_to_serialize_list(objs)
UnboundLocalError: local variable 'objs' referenced before assignment

Python 3.6.3
Django==2.1
django-fixture-magic==0.1.3

error: no such option: --query in latest pip version of django-fixture-magic

It seems django-fixture-magic is not updated after last master release. Version 0.0.5 doesn't support --query option. Here is console output:

(.virtualenv)alx@x1:~/sermonis-server$ pip install django-fixture-magic --upgrade
Downloading/unpacking django-fixture-magic from https://pypi.python.org/packages/source/d/django-fixture-magic/django-fixture-magic-0.0.5.tar.gz#md5=23ed02c964513d2a6fe5235ca49d8855
  Downloading django-fixture-magic-0.0.5.tar.gz
  Running setup.py egg_info for package django-fixture-magic

Installing collected packages: django-fixture-magic
  Found existing installation: django-fixture-magic 0.0.4
    Uninstalling django-fixture-magic:
      Successfully uninstalled django-fixture-magic
  Running setup.py install for django-fixture-magic

Successfully installed django-fixture-magic
Cleaning up...
(.virtualenv)alx@x1:~/sermonis-server$ ./manage.py dump_object sermonis.dictionary.Translation --query '{"pk__lt":100}' > 100translations.json
Usage: ./manage.py dump_object [options] <[--kitchensink | -k] object_class id1 [id2 [...]]>

Dump specific objects from the database into JSON that you can use in a fixture

./manage.py: error: no such option: --query

dump_object fails to parse command arguments correctly

The dump_object command is broken for me with Python 2.7, Django 1.9 on Windows 7 x64.

It falsely reads in the last argument as the model argument. Example:

$ python manage.py dump_object auth.User 1

will result in an options dict (see https://github.com/davedash/django-fixture-magic/blob/master/fixture_magic/management/commands/dump_object.py#L48) like:

{'ids': [],
 'kitchensink': False,
 'model': '1',
 'natural': False,
 'no_color': False,
 'pythonpath': None,
 'query': None,
 'settings': None,
 'traceback': False,
 'verbosity': 1}

As you can see, ids is empty and model got the id instead. It is always filled with the last argument for some reason. Maybe the parser behaviour changed from Python 2.6 to 2.7? I will try to find the reason.

A quick fix for me was to make the model and id arguments optional so I could do:

$ python manage.py dump_object --model=auth.User --ids=1

UnboundLocalError: local variable 'objs' referenced before assignment

Here's the trace when running dump_object with app.a_model '*' > fixture.json

dump_object with query parameter throwing JSONDecodeError

Running dump_object with query parameter gives the next exception: json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes

I'm following the declaration from README Usage section

ImportError: cannot import name loading

File "/usr/local/lib/python2.7/dist-packages/fixture_magic/management/commands/dump_object.py", line 6, in
from django.db.models import loading
ImportError: cannot import name loading

I am getting the above error. I just upgraded to django 1.9.2 and I think the problem is with django. I tried importing loading it in the shell and got the same error.

not dumping OneToOneField...

Here is my change to allow it to download OneToOneField as well.

yifanzhang@4bb06cd

query using the call_command

Could you advise how to use the dump_object command with query programatically using the call_command ?

Consider this
from django.core.management import call_command
q = UploadedFile.objects.filter(user=user).values('pk')
call_command('dump_object', 'validate.UploadedFile', '--no-follow', query=''{"pk__in": ' + str(list(q)) + '}'')

Do sent seem to work ? Any solutions ?

Custom dump nested deps

Currently if you have multiple levels of dependencies, custom_dump command only goes one level deep through straight foreign keys and not reverse foreign keys. It goes from the parent to get the children, but will only get grandchild if it can access it directly from the parent.

It would be nice to be able to use something like

CUSTOM_DUMPS = {
    'addon': {  # Initiate dump with: ./manage.py custom_dump addon id
        'primary': 'addons.addon',  # This is our reference model.
        'dependents': [  # These are the attributes/methods of the model that we wish to dump.
            'current_version',
            'current_version.files.all.0',
            {
                'primary': 'previous_versions.all',
                'dependents': ['files.all']
            }
        ],
        'order': ('app1.model1', 'app2.model2',),
        'order_cond': {'app1.model1': lambda x: 1 if x.get('fields').get('parent_model1') else 0,
                        'app2.model2': lambda x: -1 * x.get('pk')},
    }
}

Natural Keys

Is there a way to output natural keys?

Why you don't support Python3?

Do you have any plan for supporting python3 ?

Serialize ManyToMany Relationships

As I read the documentation it says it only handles FK references. I have done some serialization work in the past and have used http://code.google.com/p/wadofstuff/wiki/DjangoFullSerializers which can handle django's many2many for truly full data serialization.

Cannot install version from PyPi

Trying to install django-fixture-magic from pypi results in an error:

Downloading/unpacking django-fixture-magic (from -r requirements.txt (line 15))
  Downloading django-fixture-magic-0.0.2.macosx-10.6-i386.tar.gz
  Running setup.py egg_info for package django-fixture-magic
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
    IOError: [Errno 2] No such file or directory: '/home/evan/python-envs/plat/build/django-fixture-magic/setup.py'
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 14, in <module>

IOError: [Errno 2] No such file or directory: '/home/evan/python-envs/plat/build/django-fixture-magic/setup.py'

Installing direct from github using -e git works.

The machine is running python 2.5 on Debian Lenny.

Unable to dump BinaryField

Error:

root@hzdocker2 ~/odi # docker run -i  -v /root/odi:/code jimmy927/odi-gh:master python3.9 manage.py  dump_object core.profilejsonbinary  --query '{"person_id": 1415}'
/usr/lib/python3/dist-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.14) or chardet (3.0.4) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
Traceback (most recent call last):
  File "/code/manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/dist-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/dist-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.9/dist-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.9/dist-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.9/dist-packages/fixture_magic/management/commands/dump_object.py", line 144, in handle
    self.stdout.write(serialize(options.get('format', 'json'),
  File "/usr/local/lib/python3.9/dist-packages/django/core/serializers/__init__.py", line 129, in serialize
    s.serialize(queryset, **options)
  File "/usr/local/lib/python3.9/dist-packages/django/core/serializers/base.py", line 107, in serialize
    self.handle_field(obj, field)
  File "/usr/local/lib/python3.9/dist-packages/django/core/serializers/python.py", line 49, in handle_field
    self._current[field.name] = self._value_from_field(obj, field)
  File "/usr/local/lib/python3.9/dist-packages/django/core/serializers/python.py", line 46, in _value_from_field
    return value if is_protected_type(value) else field.value_to_string(obj)
  File "/usr/local/lib/python3.9/dist-packages/django/db/models/fields/__init__.py", line 2388, in value_to_string
    return b64encode(self.value_from_object(obj)).decode('ascii')
  File "/usr/lib/python3.9/base64.py", line 58, in b64encode
    encoded = binascii.b2a_base64(s, newline=False)
TypeError: a bytes-like object is required, not 'dict'
Waiting up to 5 seconds.
Sent all pending logs.
Sentry is attempting to send 2 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit

dump_objects shouldn't dump empty objects

I'm seeing empty PurifiedTranslations (no fields). Django doesn't like to load these.

1146, "Table 'db.table_name' doesn't exist" - error

I created a fixture using dump_object with --kitchensink option.
python manage.py dump_object --kitchensink tours.tour ids 400 > tours/fixtures/tours.json

I got the tours.json file. Now when I run tests, with the below command
./manage.py test -n It says the child_table_name does not exist

How to resolve this?

merge_fixtures should write to stdout

Just a consistency issue: dump_objects writes to stdout, while merge_fixtures doesn't.

I want to use the commands from functions, so capturing stdout is easy:

    with open("my_app/fixtures/test_fixture.json", 'w+') as f:
        call_command('dump_object', 'my_app.MyModel *list_of_some_pks, '-k', '-n', stdout=f)
        f.readlines()

Whereas trying to merge fixtures the same way fails (or I fail to capture the output):

with open("my_app/fixtures/test_fixtures_combined.json", 'w+') as f:
        call_command(
            "merge_fixtures",
            "my_app/fixtures/test_fixture_1.json",
            "my_app/fixtures/test_fixture_2.json",
            stdout=f
        )
        f.readlines()

Is this intended behaviour?

dump_object args parsing problem

Django==1.8.18
Running ./manage.py dump_object declarations.section 25976, getting error:

CommandError: object_class must be provided in the following format: app_name.model_name
Try calling dump_object with --help argument or use the following arguments:
 <[--kitchensink | -k] [--natural] [--query] object_class id [id ...]>

Investigated problem, and solved by defining named --ids argument (dump_object.py:30):

        parser.add_argument('--ids', dest='ids', default=None, nargs='*',
                            help='Use a list of ids e.g. 0 1 2 3')

merge_fixtures is having some problem parsing parameters with Python 3

Using the following Python and Django

Python 3.4.6
Django 1.10.7

When I run the merge_fixtures command

python manage.py merge_fixtures foo.json bar.json baz.json > all_fixtures.json

It shows the following error message:

usage: manage.py merge_fixtures [-h] [--version] [-v {0,1,2,3}]
                                [--settings SETTINGS]
                                [--pythonpath PYTHONPATH] [--traceback]
                                [--no-color]
manage.py merge_fixtures: error: unrecognized arguments: foo.json bar.json baz.json

I looked into it a little bit and it seems to be unhappy in parsing the command line parameters. I think it is unhappy here:

        parser = self.create_parser(argv[0], argv[1])
        options = parser.parse_args(argv[2:])

But I am a bit unfamiliar with what is expected here and didn't have time to fully investigate the issue. Do you have a hint where I should be looking?