dki-lab / freebase-setup Goto Github PK

In my downloading time,i want to record the things i have experienced these days.
I am a Chinese student in postgraduate sophomore year,and my tutor researched in KBQA.So i received a task to recurrence a paper about rng-kbqa.And it needs yours dataset called freebase.
You know China have network firewall. At first,I use a browser named 360 speed,and when i download about 20GB,it have been breaked by a reason i think about without breakpoint continuation.So i think about it once and once and thing about send a email to you for some help.And i found you are a professor in American,so i think you can't help me like upload the freebase to baidu cloud then i can download it with a high speed and have breakpoint continuation to protected my download.
And then,i find a word called wget,and i search it about some things.And now I am waiting for the download by wget command and think it will success.

你好，在下载好virtuoso.db文件后运行virtuoso.py时，程序报错：virtuoso.db/bin/virtuoso-t: not found。请问是我忽略了某些文件没有下载吗?

Hi, After I successfully started the virtuoso service, Python ran_ After preprocess. py, the relevant path information in step1. JSON is [].
For example, the data in the step1. JSON file looks like this [{"question": "what is the name of justin bieber brother", "topic_entities": ["m.06w2sn5"], "answers": ["m.0gxnnwq"]}, []]
I also placed the 'freebase rdf latest literal fixed' file in the 'virtuoso_db' folder
Do you know how to solve this problem～

I ran the "fix_freebase_literal_format.py" on the Freebase dataset and got the output file, about 31.2GB.is it correct?

Have you updated the following file recently?
https://www.dropbox.com/s/q38g0fwx1a3lz8q/virtuoso_db.zip

Hello, I followed the tutorial totally, and started the server with port 3001. However, when I opened the url, what appeared was the Baidu search interface. Is this normal？

Hi, I have followed the instructions in REAMDE, but still have problems when starting the virtuoso server.

I downloaded and unzipped virtuoso_db.zip, and confirmed the structure of my directory as follows:

.
├── LICENSE
├── README.md
├── fix_freebase_literal_format.py
├── numeric_properties.txt
├── virtuoso.py
└── virtuoso_db
    ├── virtuoso.db
    ├── virtuoso.ini
    ├── virtuoso.lck
    └── virtuoso.log

1 directory, 9 files

Then I ran the command: python3 virtuoso.py start 3001 -d virtuoso_db, the process exits after ~15 seconds, showing the following output:

RUNNING: cat /proc/meminfo | grep MemTotal | awk '{print $2}'
527485136 KB free, using 9890346.299999999 buffers, 4945173.149999999 dirty buffers
[Database]
DatabaseFile = virtuoso_db/virtuoso.db
ErrorLogFile = virtuoso_db/virtuoso.log
LockFile = virtuoso_db/virtuoso.lck
TransactionFile = virtuoso_db/virtuoso.trx
xa_persistent_file = virtuoso_db/virtuoso.pxa
ErrorLogLevel = 7
FileExtend = 200
MaxCheckpointRemap = 2000
Striping = 0
TempStorage = TempDatabase

[TempDatabase]
DatabaseFile = virtuoso_db/virtuoso-temp.db
TransactionFile = virtuoso_db/virtuoso-temp.trx
MaxCheckpointRemap = 2000
Striping = 0

[Parameters]
ServerPort = 13001
LiteMode = 0
DisableUnixSocket = 1
DisableTcpSocket = 0
ServerThreads = 100 ; increased from 20
CheckpointInterval = 60
O_DIRECT = 1 ; increased from 0
CaseMode = 2
MaxStaticCursorRows = 100000
CheckpointAuditTrail = 0
AllowOSCalls = 0
SchedulerInterval = 10
DirsAllowed = .
ThreadCleanupInterval = 0
ThreadThreshold = 10
ResourcesCleanupInterval = 0
FreeTextBatchSize = 100000
PrefixResultNames = 0
RdfFreeTextRulesSize = 100
IndexTreeMaps = 256
MaxMemPoolSize = 200000000
PrefixResultNames = 0
MacSpotlight = 0
IndexTreeMaps = 64
NumberOfBuffers = 9890346.299999999
MaxDirtyBuffers = 4945173.149999999

[SPARQL]
ResultSetMaxRows = 50000
MaxQueryCostEstimationTime = 600 ; in seconds (increased)
MaxQueryExecutionTime = 180; in seconds (increased)

[HTTPServer]
ServerPort = 3001
Charset = UTF-8
ServerThreads = 15 ; increased from unknown


virtuoso_db/virtuoso.ini
==== Starting Virtuoso server for virtuoso_db on port 3001...
RUNNING: /home/ubuntu/data/virtuoso-opensource/bin/virtuoso-t +configfile virtuoso_db/virtuoso.ini +wait
The VDBMS server process terminated prematurely
after opening the database.

Then I checked virtuoso.log, which gave the following message:

                Fri Jul 14 2023
22:46:14 OpenLink Virtuoso Universal Server
22:46:14 Version 07.20.3229-pthreads for Linux as of Aug 15 2018
22:46:14 uses parts of OpenSSL, PCRE, Html Tidy

Unfortunately, the server exits almost immediately and there isn't much informative log output.

Any help in troubleshooting this problem would be greatly appreciated.

System Information

• Ubuntu 20.04.6 LTS
• Python 3.9.16

Hi.

I downloaded both virtuoso_db.zip and virtuoso-opensource.x86_64-generic_glibc25-linux-gnu.tar.gz and unzip them.
And then I ran python3 virtuoso.py start 3001 -d virtuoso_db.

I faced out below shell output

RUNNING: cat /proc/meminfo | grep MemTotal | awk '{print $2}'
65528716 KB free, using 1228663.425 buffers, 614331.7125 dirty buffers
[Database]
DatabaseFile = virtuoso_db/virtuoso.db
ErrorLogFile = virtuoso_db/virtuoso.log
LockFile = virtuoso_db/virtuoso.lck
TransactionFile = virtuoso_db/virtuoso.trx
xa_persistent_file = virtuoso_db/virtuoso.pxa
ErrorLogLevel = 7
FileExtend = 200
MaxCheckpointRemap = 2000
Striping = 0
TempStorage = TempDatabase

[TempDatabase]
DatabaseFile = virtuoso_db/virtuoso-temp.db
TransactionFile = virtuoso_db/virtuoso-temp.trx
MaxCheckpointRemap = 2000
Striping = 0

[Parameters]
ServerPort = 13001
LiteMode = 0
DisableUnixSocket = 1
DisableTcpSocket = 0
ServerThreads = 100 ; increased from 20
CheckpointInterval = 60
O_DIRECT = 1 ; increased from 0
CaseMode = 2
MaxStaticCursorRows = 100000
CheckpointAuditTrail = 0
AllowOSCalls = 0
SchedulerInterval = 10
DirsAllowed = .
ThreadCleanupInterval = 0
ThreadThreshold = 10
ResourcesCleanupInterval = 0
FreeTextBatchSize = 100000
PrefixResultNames = 0
RdfFreeTextRulesSize = 100
IndexTreeMaps = 256
MaxMemPoolSize = 200000000
PrefixResultNames = 0
MacSpotlight = 0
IndexTreeMaps = 64
NumberOfBuffers = 1228663.425
MaxDirtyBuffers = 614331.7125

[SPARQL]
ResultSetMaxRows = 50000
MaxQueryCostEstimationTime = 600 ; in seconds (increased)
MaxQueryExecutionTime = 180; in seconds (increased)

[HTTPServer]
ServerPort = 3001
Charset = UTF-8
ServerThreads = 15 ; increased from unknown

virtuoso_db/virtuoso.ini
==== Starting Virtuoso server for virtuoso_db on port 3001...
RUNNING: {MY_PATH}/Freebase-Setup/virtuoso-opensource/bin/virtuoso-t +configfile virtuoso_db/virtuoso.ini +wait

However, I cannot open http://localhost:3001 even after waiting for 30 minutes.
I also tried to check whether the process is going or not.

$ ps -ef | grep virtuoso
iclserv+ 4047229       1  0 16:42 ?        00:00:08 {MY_PATH}/Freebase-Setup/virtuoso-opensource/bin/virtuoso-t +configfile virtuoso_db/virtuoso.ini +wait

I don't know how to solve it.
I would be grateful if someone help me.

Hey,
The following link from the README is broken:
https://www.dropbox.com/s/q38g0fwx1a3lz8q/virtuoso_db.zip
Can you provide a mirror?
Thanks!
Ohad

Virtuoso DB file can't be downloaded
wget https://www.dropbox.com/s/q38g0fwx1a3lz8q/virtuoso_db.zip

I download 3 times and i got little than 10GB file both.

So could you please share the process script ?

Hi,
Excuse me. I have downloaded your provided virtuoso.db and configured the environment. After successfully running the virtuoso.py, I use the SPARQLWrapper to query the freebase. And I find, after some query operation, the desk space occupied by the virtuoso.db file has increased from ~50G (the initial downloaded file) to ~140G. Using the command ll, I find the latest change time for this file is updated for the latest time, which means it is modified along with the using process. But I don't know why this happened? Is there some query cache? And how should I solve this problem? I hope you can help me.
I am looking forward to your reply.

Thanks！
jinhao.

Hi Su,
I read 'Useful materials about Freebase' that you put together, but I can't understand how to recognize CVT node. I wonder how to understand 'freebase.type_hints.mediator: true if a type is a CVT (Compound Value Type), Freebase's way of model Neo-Davidsonian event semantics via event reification.'
Wish for your reply！:)

Hi,

Thanks for the repo. I followed the instructions in README, downloaded the virtuoso_db and Virtuoso, and ran python3 virtuoso.py start 3001 -d virtuoso_db.

It launched successfully, the information shows ==== Starting Virtuoso server for /usr/data/virtuoso_db on port 3001.... However, when I access the server on port 3001, the webpage shows Error HTTP/1.1 404 File not found The requested URL was not found URI = '/'.

May I know how I may fix this? Thanks!!

I find it really hard to download the zip.
If I choose to download the zip from the web, the download speed is too slow, and the wget also failed and report like this:

--2023-07-06 21:49:28-- (try:20) https://www.dropbox.com/s/q38g0fwx1a3lz8q/virtuoso_db.zip
Connecting to www.dropbox.com (www.dropbox.com)|31.13.94.7|:443... failed: Unknown error.
Connecting to www.dropbox.com (www.dropbox.com)|2001::9a55:661e|:443... failed: Unknown error.
Giving up.

Thanks for sharing the codes! I have one question.

In "fix_freebase_literal_format.py", I observe that "numeric_properties.txt" is opened, but I could not find the file in this repository. Could you please upload this file, or somewhere I can get this file?

Thank you very much!
I am looking forward to receiving your comments.

Hi. Thank you so much for sharing the code.
I followed the instruction in the readme to start Virtuoso service. However, when I run the python3 fix_freebase_literal_format.py
It gives me this error.
subj, pred, obj, rest = line.split("\t")
TypeError: a bytes-like object is required, not 'str'

could you please help?

Hello, after downloading the virtuoso.db file and running virtuoso.py, the program reports an error: virtuoso.db/bin/virtuoso-t: not found. Could it be that I missed some files to download?

Hi,
Thanks for your preprocessed file! After uncompressing the file, I get the virtuoso.db. And what should i do next? It seems not the original RDF file. Thanks for your help!