dki-lab / freebase-setup Goto Github PK
View Code? Open in Web Editor NEWThe last data dump of Freebase with introductory explanation of its schema
License: Creative Commons Zero v1.0 Universal
The last data dump of Freebase with introductory explanation of its schema
License: Creative Commons Zero v1.0 Universal
In my downloading time,i want to record the things i have experienced these days.
I am a Chinese student in postgraduate sophomore year,and my tutor researched in KBQA.So i received a task to recurrence a paper about rng-kbqa.And it needs yours dataset called freebase.
You know China have network firewall. At first,I use a browser named 360 speed,and when i download about 20GB,it have been breaked by a reason i think about without breakpoint continuation.So i think about it once and once and thing about send a email to you for some help.And i found you are a professor in American,so i think you can't help me like upload the freebase to baidu cloud then i can download it with a high speed and have breakpoint continuation to protected my download.
And then,i find a word called wget,and i search it about some things.And now I am waiting for the download by wget command and think it will success.
Hi, After I successfully started the virtuoso service, Python ran_ After preprocess. py, the relevant path information in step1. JSON is [].
For example, the data in the step1. JSON file looks like this [{"question": "what is the name of justin bieber brother", "topic_entities": ["m.06w2sn5"], "answers": ["m.0gxnnwq"]}, []]
I also placed the 'freebase rdf latest literal fixed' file in the 'virtuoso_db' folder
Do you know how to solve this problem~
I ran the "fix_freebase_literal_format.py" on the Freebase dataset and got the output file, about 31.2GB.is it correct?
Have you updated the following file recently?
https://www.dropbox.com/s/q38g0fwx1a3lz8q/virtuoso_db.zip
Hello, I followed the tutorial totally, and started the server with port 3001. However, when I opened the url, what appeared was the Baidu search interface. Is this normal?
Hi, I have followed the instructions in REAMDE, but still have problems when starting the virtuoso server.
I downloaded and unzipped virtuoso_db.zip
, and confirmed the structure of my directory as follows:
.
├── LICENSE
├── README.md
├── fix_freebase_literal_format.py
├── numeric_properties.txt
├── virtuoso.py
└── virtuoso_db
├── virtuoso.db
├── virtuoso.ini
├── virtuoso.lck
└── virtuoso.log
1 directory, 9 files
Then I ran the command: python3 virtuoso.py start 3001 -d virtuoso_db
, the process exits after ~15 seconds, showing the following output:
RUNNING: cat /proc/meminfo | grep MemTotal | awk '{print $2}'
527485136 KB free, using 9890346.299999999 buffers, 4945173.149999999 dirty buffers
[Database]
DatabaseFile = virtuoso_db/virtuoso.db
ErrorLogFile = virtuoso_db/virtuoso.log
LockFile = virtuoso_db/virtuoso.lck
TransactionFile = virtuoso_db/virtuoso.trx
xa_persistent_file = virtuoso_db/virtuoso.pxa
ErrorLogLevel = 7
FileExtend = 200
MaxCheckpointRemap = 2000
Striping = 0
TempStorage = TempDatabase
[TempDatabase]
DatabaseFile = virtuoso_db/virtuoso-temp.db
TransactionFile = virtuoso_db/virtuoso-temp.trx
MaxCheckpointRemap = 2000
Striping = 0
[Parameters]
ServerPort = 13001
LiteMode = 0
DisableUnixSocket = 1
DisableTcpSocket = 0
ServerThreads = 100 ; increased from 20
CheckpointInterval = 60
O_DIRECT = 1 ; increased from 0
CaseMode = 2
MaxStaticCursorRows = 100000
CheckpointAuditTrail = 0
AllowOSCalls = 0
SchedulerInterval = 10
DirsAllowed = .
ThreadCleanupInterval = 0
ThreadThreshold = 10
ResourcesCleanupInterval = 0
FreeTextBatchSize = 100000
PrefixResultNames = 0
RdfFreeTextRulesSize = 100
IndexTreeMaps = 256
MaxMemPoolSize = 200000000
PrefixResultNames = 0
MacSpotlight = 0
IndexTreeMaps = 64
NumberOfBuffers = 9890346.299999999
MaxDirtyBuffers = 4945173.149999999
[SPARQL]
ResultSetMaxRows = 50000
MaxQueryCostEstimationTime = 600 ; in seconds (increased)
MaxQueryExecutionTime = 180; in seconds (increased)
[HTTPServer]
ServerPort = 3001
Charset = UTF-8
ServerThreads = 15 ; increased from unknown
virtuoso_db/virtuoso.ini
==== Starting Virtuoso server for virtuoso_db on port 3001...
RUNNING: /home/ubuntu/data/virtuoso-opensource/bin/virtuoso-t +configfile virtuoso_db/virtuoso.ini +wait
The VDBMS server process terminated prematurely
after opening the database.
Then I checked virtuoso.log, which gave the following message:
Fri Jul 14 2023
22:46:14 OpenLink Virtuoso Universal Server
22:46:14 Version 07.20.3229-pthreads for Linux as of Aug 15 2018
22:46:14 uses parts of OpenSSL, PCRE, Html Tidy
Unfortunately, the server exits almost immediately and there isn't much informative log output.
Any help in troubleshooting this problem would be greatly appreciated.
Hi.
I downloaded both virtuoso_db.zip
and virtuoso-opensource.x86_64-generic_glibc25-linux-gnu.tar.gz
and unzip them.
And then I ran python3 virtuoso.py start 3001 -d virtuoso_db
.
I faced out below shell output
RUNNING: cat /proc/meminfo | grep MemTotal | awk '{print $2}'
65528716 KB free, using 1228663.425 buffers, 614331.7125 dirty buffers
[Database]
DatabaseFile = virtuoso_db/virtuoso.db
ErrorLogFile = virtuoso_db/virtuoso.log
LockFile = virtuoso_db/virtuoso.lck
TransactionFile = virtuoso_db/virtuoso.trx
xa_persistent_file = virtuoso_db/virtuoso.pxa
ErrorLogLevel = 7
FileExtend = 200
MaxCheckpointRemap = 2000
Striping = 0
TempStorage = TempDatabase
[TempDatabase]
DatabaseFile = virtuoso_db/virtuoso-temp.db
TransactionFile = virtuoso_db/virtuoso-temp.trx
MaxCheckpointRemap = 2000
Striping = 0
[Parameters]
ServerPort = 13001
LiteMode = 0
DisableUnixSocket = 1
DisableTcpSocket = 0
ServerThreads = 100 ; increased from 20
CheckpointInterval = 60
O_DIRECT = 1 ; increased from 0
CaseMode = 2
MaxStaticCursorRows = 100000
CheckpointAuditTrail = 0
AllowOSCalls = 0
SchedulerInterval = 10
DirsAllowed = .
ThreadCleanupInterval = 0
ThreadThreshold = 10
ResourcesCleanupInterval = 0
FreeTextBatchSize = 100000
PrefixResultNames = 0
RdfFreeTextRulesSize = 100
IndexTreeMaps = 256
MaxMemPoolSize = 200000000
PrefixResultNames = 0
MacSpotlight = 0
IndexTreeMaps = 64
NumberOfBuffers = 1228663.425
MaxDirtyBuffers = 614331.7125
[SPARQL]
ResultSetMaxRows = 50000
MaxQueryCostEstimationTime = 600 ; in seconds (increased)
MaxQueryExecutionTime = 180; in seconds (increased)
[HTTPServer]
ServerPort = 3001
Charset = UTF-8
ServerThreads = 15 ; increased from unknown
virtuoso_db/virtuoso.ini
==== Starting Virtuoso server for virtuoso_db on port 3001...
RUNNING: {MY_PATH}/Freebase-Setup/virtuoso-opensource/bin/virtuoso-t +configfile virtuoso_db/virtuoso.ini +wait
However, I cannot open http://localhost:3001 even after waiting for 30 minutes.
I also tried to check whether the process is going or not.
$ ps -ef | grep virtuoso
iclserv+ 4047229 1 0 16:42 ? 00:00:08 {MY_PATH}/Freebase-Setup/virtuoso-opensource/bin/virtuoso-t +configfile virtuoso_db/virtuoso.ini +wait
I don't know how to solve it.
I would be grateful if someone help me.
Hey,
The following link from the README is broken:
https://www.dropbox.com/s/q38g0fwx1a3lz8q/virtuoso_db.zip
Can you provide a mirror?
Thanks!
Ohad
Virtuoso DB file can't be downloaded
wget https://www.dropbox.com/s/q38g0fwx1a3lz8q/virtuoso_db.zip
I download 3 times and i got little than 10GB file both.
So could you please share the process script ?
Hi,
Excuse me. I have downloaded your provided virtuoso.db and configured the environment. After successfully running the virtuoso.py, I use the SPARQLWrapper to query the freebase. And I find, after some query operation, the desk space occupied by the virtuoso.db file has increased from ~50G (the initial downloaded file) to ~140G. Using the command ll, I find the latest change time for this file is updated for the latest time, which means it is modified along with the using process. But I don't know why this happened? Is there some query cache? And how should I solve this problem? I hope you can help me.
I am looking forward to your reply.
Thanks!
jinhao.
Hi Su,
I read 'Useful materials about Freebase' that you put together, but I can't understand how to recognize CVT node. I wonder how to understand 'freebase.type_hints.mediator: true if a type is a CVT (Compound Value Type), Freebase's way of model Neo-Davidsonian event semantics via event reification.'
Wish for your reply!:)
Hi,
Thanks for the repo. I followed the instructions in README, downloaded the virtuoso_db and Virtuoso, and ran python3 virtuoso.py start 3001 -d virtuoso_db
.
It launched successfully, the information shows ==== Starting Virtuoso server for /usr/data/virtuoso_db on port 3001...
. However, when I access the server on port 3001, the webpage shows Error HTTP/1.1 404 File not found The requested URL was not found URI = '/'
.
May I know how I may fix this? Thanks!!
I find it really hard to download the zip.
If I choose to download the zip from the web, the download speed is too slow, and the wget also failed and report like this:
--2023-07-06 21:49:28-- (try:20) https://www.dropbox.com/s/q38g0fwx1a3lz8q/virtuoso_db.zip
Connecting to www.dropbox.com (www.dropbox.com)|31.13.94.7|:443... failed: Unknown error.
Connecting to www.dropbox.com (www.dropbox.com)|2001::9a55:661e|:443... failed: Unknown error.
Giving up.
Thanks for sharing the codes! I have one question.
In "fix_freebase_literal_format.py", I observe that "numeric_properties.txt" is opened, but I could not find the file in this repository. Could you please upload this file, or somewhere I can get this file?
Thank you very much!
I am looking forward to receiving your comments.
Hi. Thank you so much for sharing the code.
I followed the instruction in the readme to start Virtuoso service. However, when I run the python3 fix_freebase_literal_format.py
It gives me this error.
subj, pred, obj, rest = line.split("\t")
TypeError: a bytes-like object is required, not 'str'
could you please help?
Hi,
Thanks for your preprocessed file! After uncompressing the file, I get the virtuoso.db. And what should i do next? It seems not the original RDF file. Thanks for your help!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.