My name is Luca Pette (He/Him), I'm an polyglot developer based in Berlin ๐ฉ๐ช.
Available for hire.
CLI utility for fake data generation
License: MIT License
My name is Luca Pette (He/Him), I'm an polyglot developer based in Berlin ๐ฉ๐ช.
Available for hire.
Currently, when the data is output as SQL, the SQL statement is not escaped properly, like the quotation mark '
and comment --
in the string.
Example:
INSERT INTO DUMMY_DATA_10000 (email,name,animal,address_state,username,occupation,noun,city,phone,industry,adjectives) VALUES ('[email protected]','Waylon Padilla','platypus','Delaware','mrmartineau','cosmetologist','servitude','Chesterfield','+6929226895233','Mining & Metals','evergreen');
INSERT INTO DUMMY_DATA_10000 (email,name,animal,address_state,username,occupation,noun,city,phone,industry,adjectives) VALUES ('[email protected]','Rolland Medina','wildcat','North Dakota','jesseddy','administrative assistant','ma'am','Alameda','+2256640101810','Environmental Services','walk-on');
Hey, I just discovered this project and gave it a try. Naturally I tried to run fakedata help
because most Go programs I used have this as a command; fakedata
without an argument however just outputs 10 empty lines which is an odd behaviour IMO.
I looked at the code and this happens because the limit
flag defaults to 10 and the fakedata generation runs with the default seperator (space), so 10 "empty datasets" are printed.
I think it would be a good idea to have Unknown command: X
message for unknown commands. I see that you check for empty args here.
If you want to I could give this a shot, tho I am quite new to Go. If it's intended behaviour I guess this issue can just be closed. ๐
Right now the following template will always produce a sequence of 5 numbers:
echo '{{ range Loop 5 }}{{ Int 0 9 }}{{ end }}' | fakedata -l 5
36513
62260
38785
18363
82136
It'd be great to have an ability to specify a lower and higher bounds, so that range Loop x y
would generate a sequence of random length:
echo '{{ range Loop 2 7 }}{{ Int 0 9 }}{{ end }}' | fakedata -l 5
365
6226013
3878
183635
82
On each iteration specified by -l 5
, {{ range Loop x y }}
would produce a random number between x
and y
, inclusive, and loop that many times.
As fakedata
is a CLI tool, auto-completion would make usage much more enjoyable. I suggest we provide an option that outputs the completion function. Something like:
$ fakedata --completion=bash
so that we can ask users to add eval $(fakedata --completion)
(I'm not sure it's the best API, suggestions, as always, more than welcome).
While doing some quick search around the topic I run into complete but I'm not sure it suits this use case yet.
I'm thinking support for ndjson would be nice for practical purposes:
$ fakedata -f ndjson uuidv4 email
{"uuidv4": "38c3354d-7691-4262-8857-6cc163eadb36", "email": "[email protected]"}
{"uuidv4": "070cc2ae-e336-4cd9-b229-b55df063e050", "email": "[email protected]"}
{"uuidv4": "4c809f62-bd3c-4bb1-bfe0-acec390e9215", "email": "[email protected]"}
{"uuidv4": "6e89f983-3889-4021-b82a-7bdf76720d08", "email": "[email protected]"}
{"uuidv4": "42182a57-b3f5-4c61-9e31-49cd2541e34a", "email": "[email protected]"}
{"uuidv4": "85a63dc2-256d-4f52-ab25-5fce441fa5d7", "email": "[email protected]"}
{"uuidv4": "9bfb6868-e8ae-44d7-988b-ad50ae261217", "email": "[email protected]"}
{"uuidv4": "ce1a8a7d-2c01-4551-b478-d5f082664143", "email": "[email protected]"}
{"uuidv4": "6c28d066-91e5-4186-9767-c5e3371d5407", "email": "[email protected]"}
{"uuidv4": "98a05536-77bb-4a4e-994a-708a82529d2b", "email": "[email protected]"}
it would use the same convention as the SQL formatter to rename fields:
$ fakedata -f ndjson uuidv4:id email
{"id": "f74d7686-2027-4cdc-bddf-cc22804a03e7", "email": "[email protected]"}
{"id": "f5db0ea3-9056-4dd8-87da-d258aa91c82a", "email": "[email protected]"}
{"id": "fad39788-0290-49fd-ba34-8fc0f3e25f71", "email": "[email protected]"}
{"id": "359c2791-b95a-4fab-a552-f08c22ff5c34", "email": "[email protected]"}
{"id": "fd797bfb-c281-4093-ac77-cb10e5dd5af3", "email": "[email protected]"}
{"id": "d27e050a-43f2-4586-b03e-37a09942eb0a", "email": "[email protected]"}
{"id": "23715de6-4ae9-4a3e-861e-aad063b38f48", "email": "[email protected]"}
{"id": "bedd7dab-f51c-4f8f-9658-c9bc1164551c", "email": "[email protected]"}
{"id": "80378f51-3cf4-4bb8-9f41-b73ee6b38240", "email": "[email protected]"}
{"id": "06954001-ab7b-4307-a014-c497727ac1ab", "email": "[email protected]"}
I would not try to support nested JSON because there's not a nice syntax for that and templates do support the use case already
In other Go projects, I'm enjoying having a package pkg/testutil
with test helpers. It helps polishing the testing code and encourages more testing (which is hardly a bad thing). To get it started we could introduce a testutil.Diff(a, b interface{}) err
and use it wherever we make use of github.com/kr/pretty
. Right now we use it in the integration tests only but if we have it in a package then we can use it in the unit tests too.
Right now, we do not check if the input format is known. That leads to a strange interaction:
$ fakedata -f=unknown email
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
We default to the space separated format, therefore the current behavior. It's rather odd (and hard to notice) if the command has a small typo. Example:
$ fakedata -f=tav -l=1 email ipv4
[email protected] 239.201.152.149
The output contains spaces instead of tabs because the input format was tav
.
Having flags as global variable isn't a good practice and I've been wanting to move them for a while
grep -v /vendor
from makefile t.Helper()
at the start of all test helpers1.9.x
requirementAs specified here. We can (should?) support the multiple versions available and follow the naming of the ip generators: uuidv1
, uuidv2
and so on.
For this generator, it may make sense to do some preliminary research to find a library we're comfortable with to add as a dependency.
As specified here. I think it'd make sense to accept a range of dates but I'm not sure yet about what's the best format. Right now we parse dates in the format YYYY-MM-DD
which is a good starting point but it doesn't contain time information.
Based on this conversation
When reading values from a file, if the last line ends with \n
(which is often the case), and empty value is generated.
For instance:
$ printf 'a\nb\nc' >| a
$ cat -A a
a$
b$
c
$ fakedata -l 10 file:a
a
b
b
a
c
b
a
c
a
a
but
$ printf 'a\nb\nc\n' >| a
$ cat -A a
a$
b$
c$
$ fakedata -l 10 file:a
a
b
a
b
c
a
c
The file can't therefore be generated using another command. For instance
$ find . -name '*.txt' > textfiles.dat
$ fakedata -l 10 file:textfiles.dat
since every such command (or even editor), and fakedata
itself, add a \n
on the last line.
I didn't know about the custom sql column names when I implemented the key validation! While I check for ,
inside the key, I do not check for =
and therefore the following command fails validation.
$ fakedata --format sql --limit 1 login=email referral=domain
Unknown generator: login=email.Unknown generator: referral=domain.
See fakedata --generators for a list of available generators.
The error is also no longer multilined.
Since there's two ways of declaring the syntax I think it might be best to extract the parameter "parsing" into its own function, maybe this function could be shared with the rest of the code? This way there'd be only one spot where parameter splitting/"parsing" is defined and done.
fakedata
template specsThis issue describes the upcoming template feature for
fakedata
and is the basis of discussion around implementation
Despite being powerful already, fakedata
will benefit from a template mechanism. With the underlying Go template engine we can implement a easy-to-use template mechanism which allows users to generate fake datasets in all kinds of formats, regardless of build-in formatters like sql
, csv
, et all.
Go Templates allow to combine values with printf
so we get a way to combine generators our of the box, e.g. {{ printf "%s, %s" Lastname Firstname }}
to print out names in an alternative format to what we are used to in Germany (and other countries).
In order to support templates we need to add a new "top level" command to fakedata
which can accomplish the following tasks:
Stdout
or to a fileBy introducing a new top level command we avoid interference with existing generators and separate this functionality from the "direct" generator invocation.
All generators are available as template functions and can be used inside the template to generate data as seen below:
{{ Name }} <{{ Email }}>
The example above will generate Name and E-Mail pairs like so:
Jane Doe <[email protected]>
Jon Doe <[email protected]>
As mentioned above we can now also combine generators inside the templates which gives a huge advantage over using generators in a "as-is" way. Users can define their own format or create specific datasets if needed.
Go templates have a FuncMap
which is a mapping of functions available inside the parsed template. We either need to manually add all generators or find a way to automatically make them available.
I would like to propose the addition of functions like Last
, First
, Odd
, and Even
. I believe these "little helpers" can provide a great interface when generating data, e.g. for adding a final semicolon to the end of an SQL statement or to close a JSON block properly.
{
{{ range . }}
{
"name": "{{ Name }}",
"email": "{{ Email }}",
} {{ if not Last }},{{end}}
{{ end }}
}
Running the template generator on the template will create a file with valid JSON as seen below:
{
{
"name": "Kevin",
"email": "[email protected]"
},
{
"name": "Emma",
"email": "[email protected]"
}
}
So this is my proposal written off the top of my head. Are there any things I missed? Feel free to add your input, I believe I will start work on Friday or next week (Monday/Tuesday).
https://github.com/dariusk/corpora/blob/master/data/geography/us_cities.json I thought we were doing this already
As of today, we declare generator in a bit of an unusual way. Conversation started here
fakedata is working great so far! Thanks for creating it!
Just a small thing to add:
I'm using it to generate some SQL. One column should be the same for all rows.
Right now I pass id=int,12345..12345
to limit the range to only this number.
Would be nice if I could also just do id=int,12345
.
Also, if I like to do the same thing for a string value, there is currently no way to do so, right?
I think it makes sense to have more top-level domains added to the generator, especially the new once like .codes
, .academy
, .website
, etc.
Depending on what you're using the fakedata for it's good to have the diversity of the new, longer tlds. The old domain validation patterns (e.g. checking for length 2-4 characters) don't work on new tlds so having them in fakedata could provide a benefit when testing large datasets against an API or validation implementation.
Based on a conversation we had on slack. @jorinvo proposed we change the way constrains/range (we should probably just call these "options" so that it's generic enough to work for different generators) this way:
$ fakedata user=enum,John..Joe..James age=int,1..100 # current syntax
$
$
$ fakedata user=enum:John,Joe,James age=int:1,100 # new syntax
I think it looks great and I'd proceed right away. The feature requires us to:
It's a very exciting change as it leads to a much simpler (therefore better UI). I can take care of it unless any of you would like to give it a try.
I know the name generators do only english names, but i wanted to generate a hungarian name order with "lastname firstname" into one field, and found out that this is currently not possible.
This also leads to a possible new feature, that a language specific dict map may be loaded run-time with a parameter :) But let me go back to the original idea
The enum function allows freely definiable datasets, but sometimes it would be desired to combine at least two fileds into one, with a separator (that may also be empty)
I am in no means a programmer, but can read code and found there are working examples that already do this kind of data generation:
domain => domain.name,domain.tld,"."
email => username,domain,"@"
name => name.first,name.last," "
Maybe a generator similar to enum could implement the withSep function as a field definition, that is able to understand the 3 parameters, but the splitting of the parameters would need a bit thinking...
Anyway, thank you for this great tool, and that you in advance, if you find the time and motivation to work on this feature.
#12 makes me think that it already time to introduce some higher level integration tests. I'd like to test the output of the binary but I never did this with Go so any suggestion is more than welcome. I remember reading somewhere that terraform has this kind of test. I'll do some research and report back here
As per title.
I suppose it could have a UI similar to #79. Meaning:
fakedata address
# generates international addresses
and
fakedata address.[country-code]
# generates country specific addresses
I want to see the description of the date generator and it gives me a runtime error!
Use Version v1.1.0
of fakedata installed with brew install lucapette/tap/fakedata
on macOS 10.12.6.
Type fakedata -g date
into a Terminal.
The description of the date generator
The following was printed in the Terminal:
Description: random date in the format YYYY-MM-DD. By default, it generates dates in the last year
Example:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x112b8ad]
goroutine 1 [running]:
main.main()
/Users/lucapette/src/github.com/lucapette/fakedata/main.go:127 +0x9dd
#43 is an exciting change to fakedata
UI but, as @KevinGimbel pointed out, it's a breaking change and we agreed it's a good idea to aim for a first stable version.
Here is what i have in mind at the moment:
Completing all the features/fixes of this list should give us a pretty solid (and nicely featured) fakedata v1.0.0
. I'll be tagging the existing issues accordingly so we get a better overview of what's done and what needs to be done.
About #17 and #30, I know @KevinGimbel and @jorinvo respectively offered to help with those. Both features aren't easy to implement so I'm thinking I can take care of the rest while we shape those two complicated features.
Suggestions about what to add/remove from v1.0.0
are always more than welcome!
Here is the up to date milestone v1.0.0
I'm facing a difficulty when generating JSON arrays by means of templates.
In the following example it's not possible to generate a valid JSON array, because of the trailing coma:
echo '[{{ range Loop 5 }}"{{ Int }}",{{ end }}]' | fakedata -l 1
["52","419","365","561","753",]
If only there'd be a way to concat items before outputting them. Something like this would work great:
echo '[{{ range Loop 5 "," }}"{{ Int }}",{{ end }}]' | fakedata -l 1
["52","419","365","561","753"]
Above, by specifying another argument in range
, it'd be possible to define a string that would be used when concatenating items before final output.
Hello!
i follow the readme
i entered this command echo {{ Enum "100" "2020" "40040" }} | fakedata -l 50 > test.txt
But there is an error like this "template: template:1: expected :="
how to fix it?
$ fakedata email username
[email protected] _shahedk
[email protected] tgerken
[email protected] 91bilal
[email protected] blakesimkins
[email protected] areandacom
[email protected] rmlewisuk
[email protected] robergd
[email protected] charliecwaite
[email protected] baluli
[email protected] weglov
vs
$ fakedata --header email username
email username
[email protected] _shahedk
[email protected] tgerken
[email protected] 91bilal
[email protected] blakesimkins
[email protected] areandacom
[email protected] rmlewisuk
[email protected] robergd
[email protected] charliecwaite
[email protected] baluli
[email protected] weglov
It should also support generator renaming (we use it in the SQL formatter)
dictionary
could return a random word from an English dictionary (or a top 1000 words list).
Along with this there could be dictionary.noun
, dictionary.verb
, dictionary.adjective
.
So far we never had a conversation about which generators make sense for version 1.0.0 and while working on #50 I realised it's a good time to change that. Right now most of what we did is either inherited from phony or imported (as an example) from corpora. In practise, we have the following generators:
animal animal breed
animal.cat random cat breed
color one word color
country Full country name
country.code 2-digit country code
date random date in the format YYYY-MM-DD. By default, it generates dates in the last year
domain domain
domain.tld valid TLD name from https://data.iana.org/TLD/tlds-alpha-by-domain.txt
double double number
email email
emoji emoji from https://github.com/dariusk/corpora/blob/master/data/words/emojis.json
enum value from an enum. By default, the enum is foo,bar,baz. It accepts a list of comma-separated values
event.action clicked|purchased|viewed|watched
file random value from a file. It accepts a file path. It can be either relative or absolute. The file must contain a value per line
http.method DELETE|GET|HEAD|OPTION|PATCH|POST|PUT
int positive integer between 1 and 1000
ipv4 ipv4
ipv6 ipv6
latitude latitude
longitude longitude
mac.address mac address
name name.first + " " + name.last
name.first capitalized first name
name.last capitalized last name
noun noun from https://github.com/dariusk/corpora/blob/master/data/words/nouns.json
product.category Beauty|Games|Movies|Tools|..
product.name invented product name
state Full US state name
state.code 2-digit US state name
timezone tz in the form Area/City
username username using the pattern \w+
I would like to remove the following:
product.category
product.name
and add:
animal.dog
sentence https://github.com/dariusk/corpora/blob/master/data/words/harvard_sentences.json
industry https://github.com/dariusk/corpora/blob/master/data/corporations/industries.json
occupation https://github.com/dariusk/corpora/blob/master/data/humans/occupations.json
At first, I thought we should remove more (and commented in #44 about that) but looking at the data we can import from corpora I realise it's probably nicer if people can choose what to generate even more granularly.
It would immensely help to have an ability to define custom helpers in templates. Custom helpers could be used across the template by referencing the custom helper name anywhere in the template.
Helper specification:
a helper definition would look like this:
{{ define "my_helper" }}
Any string including references to core helpers like {{ Int }} or {{ other_custom_helper }} defined earlier
{{ end }}
it would be defined directly in the template:
{{ define "my_buddy" }}
{{ NameFirst }} {{ NameLast }}
{{ end }}
Hello, {{ my_buddy }}!
when rendered, the helper itself would resolve to an empty string: when custom helper is defined in a template file, and that template file is passed to fakedata
, all symbols occupied by all custom helpers would shrink into empty strings after initial parsing / interpretation phase.
Consider the following template:
{{ define "my_buddy" }}
{{ NameFirst }} {{ NameLast }}
{{ end }}
{{ define "the_phone_number" }}
"+380{{ Enum "97" "63" "50" }}{{ range Loop 7 }}{{ Int 0 9 }}{{end}}"
{{ end }}
The phone number of my buddy {{ my_buddy }} is {{ the_phone_number }}.
Passing it to fakedata
via:
cat template.tmpl | fakedata -l 1
Would result in the following sample output:
The phone number of my buddy Brock Ebonie is +380974567313.
Note the two blank lines. I'd be great to have them removed as well, but I'm not sure if it's possible to do this correctly.
it might be better (but also probably harder) to have the custom functions defined in an external file, and have it loaded before executing a template:
echo "My phone number is {{ the_phone_number }}." | fakedata -x my_functions.tmpl -l 1
I'm working on a template that is a 300 lines JSON document. It looks like this (reduced significantly, for brevity):
{
"person": {
"contact_info": {
"phone_number": "+380637274547"
},
"confident_persons": [
{
"contact_info": {
"phone_number": "+380977557274"
}
},
{
"contact_info": {
"phone_number": "+380503311437"
}
}
]
}
}
For it, I've made the following template:
{
"person": {
"contact_info": {
"phone_number": "+380{{ Enum "97" "63" "50" }}{{ range Loop 7 }}{{ Int 0 9 }}{{end}}"
},
"confident_persons": [
{
"contact_info": {
"phone_number": "+380{{ Enum "97" "63" "50" }}{{ range Loop 7 }}{{ Int 0 9 }}{{end}}"
}
},
{
"contact_info": {
"phone_number": "+380{{ Enum "97" "63" "50" }}{{ range Loop 7 }}{{ Int 0 9 }}{{end}}"
}
}
]
}
}
Obviously there's a huge duplication going on here. By defining a custom helper function, this could be reduced significantly, to:
{{ define "ukrainian_phone" }}
+380{{ Enum "97" "63" "50" }}{{ range Loop 7 }}{{ Int 0 9 }}{{end}}
{{ end }}
{
"person": {
"contact_info": {
"phone_number": "{{ ukrainian_phone }}"
},
"confident_persons": [
{
"contact_info": {
"phone_number": "{{ ukrainian_phone }}"
}
},
{
"contact_info": {
"phone_number": "{{ ukrainian_phone }}"
}
}
]
}
}
Hello it would be nice to have such generator too.
https://en.wikipedia.org/wiki/Lorem_ipsum
~ โ fakedata language_id=int,1..1
panic: invalid argument to Intn
goroutine 1 [running]:
math/rand.(*Rand).Intn(0xc42001a240, 0x0, 0x0)
/usr/local/Cellar/go/1.8.1/libexec/src/math/rand/rand.go:141 +0xc7
math/rand.Intn(0x0, 0x1)
/usr/local/Cellar/go/1.8.1/libexec/src/math/rand/rand.go:276 +0x37
github.com/lucapette/fakedata/pkg/fakedata.integer.func1(0x7fff5fbff891, 0xb, 0x7fff5fbff89d, 0x3, 0x7fff5fbff8a1, 0x1, 0x7fff5fbff8a4, 0x1, 0x10df880, 0xc420051d01)
/Users/lucapette/src/github.com/lucapette/fakedata/pkg/fakedata/generator.go:159 +0x6c
github.com/lucapette/fakedata/pkg/fakedata.generate(0x7fff5fbff891, 0xb, 0x7fff5fbff89d, 0x3, 0x7fff5fbff8a1, 0x1, 0x7fff5fbff8a4, 0x1, 0xc4200168b0, 0x10)
/Users/lucapette/src/github.com/lucapette/fakedata/pkg/fakedata/generator.go:28 +0xb4
github.com/lucapette/fakedata/pkg/fakedata.GenerateRow(0xc420012440, 0x1, 0x1, 0x1192300, 0xc4200168b0, 0x1, 0xc42000c310)
/Users/lucapette/src/github.com/lucapette/fakedata/pkg/fakedata/fakedata.go:14 +0x114
main.main()
/Users/lucapette/src/github.com/lucapette/fakedata/cmd/fakedata.go:95 +0x22e
Thanks to @jorinvo we now have a slack channel and it's worth mentioning in the README so people that have high interest in the conversation can join
fakedata
may or may not have some issues working with Windows and the Windows file system (e.g. file
generator) due to it being developed and tested on *nix only (MacOS, Linux).
As a first step we should run some basic tests on a Windows machine.
TODO
fakedata
on windowsfile
Right now, we have only a date
generator for the format YYYY-MM-DD
. I'd use this issue to collect more formats and (probably?) add more date formats all at once.
The discussion started here. Thanks to @gnanet to bring this up.
The first approach that comes to mind is to use a small convention based on the name of the input file (without the extension) for the generator name, moreover I think it makes sense that this particular option is repeatable. For example:
$ echo "ciao\nhello" > file1.txt
$ fakedata --from-file=file1.txt -l=2 email file1
ciao [email protected]
hello [email protected]
$
$ echo "priviet\nhallo" >file2.txt
$ fakedata --from-file=file2.txt --from-file=file1.txt -l=2 file1 file2
ciao priviet
hello hallo
I like it! And it's a fun feature to build :)
@gnagnet does this work for what you had in mind?
It will be great to have this on AUR. I can give this a shot if you want.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.