Comments (10)
So, after a lot of research, trial and error, we've reached that crocodoc isn't useful for us. So what we'll do is the following:
- Use either the OpenOffice command line tool or unoconv to convert Despachos from .doc/.docx to HTML, scrapable files (i'd rather we used unoconv since it supports both OpenOffice and LibreOffice formats, and it's a specific command-line tool).
- Scrape the resulting HTML Despacho with noodle into a DemocracyOS-compatible JSON structure.
- Persist the resulting JSON in mongo.
There you go, @ultraklon
from bill-scraper.
I finally made this work with Libre Office, using the following command line
soffice --headless --convert-to htm:html --outdir ./ despacho.doc
Nothing special, I just tried it again, seems to need the program (LibreOffice) to be closed for this to work
from bill-scraper.
Will this work server side?
from bill-scraper.
I don't know what server are we using. What server are we using?
from bill-scraper.
I mean, we need a place to have a copy of LibreOffice and execute it somehow with parameters and disk access
from bill-scraper.
No, we can't guarantee to have disk access in a server running DemocracyOS.
For the time being make it be HTML and scrape it locally. Even if we have to upload the resulting JSON by hand, I'd rather have that than no process at all. We could then add a script that does it all from a single command.
I guess we could upload the resulting HTML to an accesible URL (GDrive, Dropbox, whatever) and have the scraper run server side. Still, following @jazzido's advice, it's better to have it run locally, halting on ALL errors and ensuring things get scraped the right way, than trusting too much on an automated solution that fails silently and messes up our data.
from bill-scraper.
Anyway
from bill-scraper.
Got it, I'm thinking about using GDrive API to convert docs.
I'll proceed with Noodle, w/o caring about how to receibe html docs (yet)
from bill-scraper.
@ultraklon , @oscarguindzberg will be uploading newly obtained data files to @DemocracyOS 's beta app, he'll be getting in touch with you for some assistance.
BTW: where's the code for converting .docs to .htmls through Google's GDrive API?
from bill-scraper.
I'm closing this as it's now followed by #10
from bill-scraper.
Related Issues (12)
- Regex discards important text HOT 1
- Import "Despachos" into app
- Law category filter HOT 1
- User options drop-down HOT 1
- Encoding to UTF-8 HOT 1
- Property names in english HOT 3
- Preserve article number HOT 8
- Preserve bill number and Id HOT 3
- Parser says Error parsing: SyntaxError: Unexpected token
- How should variables be passed? HOT 4
- Matching current info with existent structure HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bill-scraper.