Comments (6)
Hi,
Yes, I plan to clean up and release those scripts, but I'm too busy to do it now, will probably get it done later this year.
from stog.
hi,may I ask have this issue solved?
from stog.
Here's some code to generate amr_utils from your training set. Note that you will need to download a few things for this to work (see comments in the code). I can verify this runs and produces the needed files but I'm not certain about any compatibility with different versions of the downloaded files.
#!/usr/bin/python3
import os
from types import SimpleNamespace
from stog.data.dataset_readers.amr_parsing.preprocess.recategorizer import Recategorizer
from stog.data.dataset_readers.amr_parsing.node_utils import NodeUtilities
# amr_utils can be downloaded from https://www.cs.jhu.edu/~s.zhang/data/AMR/amr_2.0_utils.tar.gz
# Inside the tar.gz are 2 files (joints.txt, text_anonymization_rules.json) that I'm no sure where
# the originate from but are used in STOG
if __name__ == '__main__':
util_dir = 'data/amr_utils'
data_dir = 'data/LDCProcess'
train_data = os.path.join(data_dir, 'train.txt.features')
propbank_dir = 'data/AMR-Downloads/propbank-frames-2018-04-20/'
verbal_fn = 'data/AMR-Downloads/verbalization-list-v1.06.txt'
assert os.path.exists(train_data), 'You must annotate the LDC training data before running this script'
# Create:
# entity_type_cooccur_counter.json, name_op_cooccur_counter.json, name_type_cooccur_counter.json,
# wiki_span_cooccur_counter.json
recategorizer = Recategorizer(train_data, build_utils=True, util_dir=util_dir)
# Creates: lemma_frame_counter.json, frame_lemma_counter.json, senseless_node_counter.json
# Need to download verbalization-list-v1.06.txt from https://amr.isi.edu/download.html
# wget https://amr.isi.edu/download/lists/verbalization-list-v1.06.txt
# Also download probank frames from https://github.com/propbank/propbank-frames/
# The 2018-04-20 is the lastest as of 2020/05/24. Originally the directory was tagged
# at propbank-frames-xml-2016-03-08. There are a few releases on the site that might be earlier
args = SimpleNamespace()
args.amr_train_files = [train_data]
args.propbank_dir = propbank_dir
args.verbalization_file = verbal_fn
args.dump_dir = util_dir
args.train_file_base_freq = 1.0
args.propbank_base_freq = 1.0
args.propbank_bonus = 10.0
args.verbalization_base_freq= 1.0
args.verbalize_freq = 100.0
args.maybe_verbalize_freq = 100.0
args.verbalize_bonus = 10.0
nu = NodeUtilities.from_raw(
args.amr_train_files,
args.propbank_dir,
args.verbalization_file,
args.dump_dir,
args.train_file_base_freq,
args.propbank_base_freq,
args.propbank_bonus,
args.verbalization_base_freq,
args.verbalize_freq,
args.maybe_verbalize_freq,
args.verbalize_bonus)
from stog.
Here's some code to generate amr_utils from your training set. Note that you will need to download a few things for this to work (see comments in the code). I can verify this runs and produces the needed files but I'm not certain about any compatibility with different versions of the downloaded files.
amr_utils can be downloaded from https://www.cs.jhu.edu/~s.zhang/data/AMR/amr_2.0_utils.tar.gz
Inside the tar.gz are 2 files (joints.txt, text_anonymization_rules.json) that I'm no sure where
the originate from but are used in STOG
Hi, @bjascob I think I find the originate of joints.txt, which is from https://github.com/ChunchuanLv/AMR_AS_GRAPH_PREDICTION/blob/master/data/joints.txt.
And I wonder if you know the originate of text_anonymization_rules.json or the script to generate it now?
from stog.
Sorry, no idea. You'll have to ask the author.
from stog.
Hi, I find the text_anonymization_rules.json
is related to the Anonymized process.
It seems that text_anonymization_rules.json
in amr_2.0_utils only contains Entities on AMR 2.0 .
Could you share with me the script to generate it for my own datasets? Thanks!
from stog.
Related Issues (20)
- Question Regarding GPU-requirement HOT 2
- Type-o in README
- penman.EncodeError: Invalid graph; possibly disconnected. HOT 2
- How to generate AMR from a file HOT 5
- bug
- details of the code
- Occasional problem with training on subsets of AMR 2.0
- how to parsing the DailyDialog
- Training on AMR 3.0 (LDC2020T02) HOT 1
- how to generate the AMR based on other dataset like WMT14 or iwslt2014 ,etc.
- Are the source-side and target-side vocabularies shared?
- Progress 20210824
- Add Glove Indo and FastText to artifact
- Using pre-trained models
- evaluation.sh requires python2 HOT 1
- How to predict AMR graph for arbitrary sentences? HOT 10
- Prediction with only one input sentence fails HOT 1
- Pretrained Model HOT 1
- Pin Penman version in requirements HOT 1
- Error of preprocessing with CoreNLP HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from stog.