kmcelwee / fortune-100-blm-report

Analysis of Fortune 100 twitter data

Fortune 100 BLM Report

An analysis of fortune-100-blm-dataset. Link to the website: https://www.kmcelwee.com/fortune-100-blm-report/site/

• docs: a poorly-named directory for JSON data used in the site
• figures/tweet-histograms: collection of static tweet histograms useful for debugging
• fortune-100-blm-dataset: the submodule used to inform analysis.
• scripts: parsers for generating JSON and querying Twitter.
• supplementary-data: data brought in from external sources to enrich the BLM dataset.
• site: where the actual site is hosted.

fortune-100-blm-report's Issues

Double check every tweet

After finishing data visualization, run a final pass through all tweets to ensure consistency

Wrap report better

Container within the report to hold paragraphs and headers? Otherwise it's just explicitly declaring the width of headers and paragraphs and images

Summarize "BLM" by sector

Error viewing ID 1266171968728043520 in oembed

Include it in error list for twitter-oembeds.json, rewrite logic to include. Also research what this used to be.

Standardize all column names

all lowercase, spaces replaced with underscore

Convert IDs to strings, browser can't handle large ints.

IDs are rounded when requested in D3 (or at least in the browser), making them useless (e.g. 1272562122258604032 -> 1272562122258604000). They need to be stored as strings.

Create legend

This isn't straightforward in D3. Not sure how to best move forward

Add fortune 100 rank to data visualization

Summarize full history of #BLM

Spike implications of "Twitter for Advertisers"

https://twitter.com/CocaCola/status/1268235977614655488

These tweets do not show up in either search or scrape. I found this through a CocaCola reply. There's a lot to go through with this exception

Ensure data visualizations are responsive

https://medium.com/@louisemoxy/a-simple-way-to-make-d3-js-charts-svgs-responsive-7afb04bc2e4b

Make corporate visualizations own id

Edit HTML s/t the ids don't need to be rendered by D3. This may allow the redirect to jump directly to the desired visualization

Include replies to self

Spike on implications of slightly altering reply / thread filtering.

https://twitter.com/pfizer/status/1274002179368652807

Separate report and corporate summaries

Create an index.html with the report and then a all-corporations.html for summaries

Publish profiles of each quiet corporation

Spike on publishing options

Isn't combining markdown with charts what Jupyter was made for? But I want a friendlier medium to any reader... so a proper website? I don't want to invest so much time in that! I could generate a website from Markdown which points to the files generated here. The only concern is interactive graphs, but D3 has certainly been used in GitHub pages elsewhere.

Include max_count in histogram JSOn

This calculation is not straightforward in JS. We can just include it directly in the histogram.json

Add "Juneteenth", "BLM", etc. tags

Use Bootstrap tags to insert into visualization on hover.

Create script to search each companies entire history for "BLM", "Black Lives Matter", or "#BlackLivesMatter"

Create interactive tweet histograms with hover as twitter embed

On the project's index.html, list all companies and their histograms. First just get it done with simple D3, then try to implement hover as twitter embed. Place data files in a docs dir

Add a "Last Updated" item to corporate summaries

Add non-twitter-using companies to data viz

Add a brief section about companies without twitter accounts

Make gists and link for each calculation

a la The Correspondent. Span with invisible content attached to word. On click span becomes block that contains content.

Include search functionality

https://observablehq.com/@jashkenas/inputs @ autoSelect

Address oembed error with tweet ID 1280186066469822464

Slim down oembed JSON

Only need HTML. We can also strip out the script widge js`

Highlight by tag

Create an interface that highlights (with large yellow circle?) the tweets with the selected tags

Write generate script for weekly fortune 100 tweets overview

Spike: How to make tweet histograms larger

We have both the static images and the interactive visualizations. For static images, we can place the legend above the histogram instead of to the right. For dynamic images, see how we can increase circle size, reduce margins, and ensure that the graph is taking up the full given width. Also consider widening default text block width.

Introduce figcaptions

Explore what new questions might be answered by Twitter API v2

Rounding errors found in IDs

Oembed process revealed IDs that had been rounded:

1265004698148495400
1265812305817862100
1266080125805834200

This exists within the dataset in fortune-100-blm-dataset. I believe I manually entered values for Lowe's because of API limits, so that might be what's happening.

• Request oembed for all tweets ending in 00 to double check that it's limited to these tweets.
• Dig into fortune-100-blm-dataset repo and double check scripts.
• Pandas automatically reads the ID column as an integer. Research that this doesn't cause issues.

Ensure that #blacklivesmatter search is not case-sensitive

Intel tweeted #blacklivesmatter but I may have not looked properly

Experiment: Place all notes material after the paragraph

Design some kind of block after each paragraph where we can place images, gists, links, etc.

Spike options for archiving citations

Request some kind of web archive?

Write script for "first tweet" histogram

This could be either static or D3. Maybe just static for now.

Import protest data and graph

Shift all right-aligned text to left-align

Improve OEmbed error handling

• Create a file reviewed-oembed-errors.txt that lists IDs reviewed in #14.
• Update collect-twitter-oembeds.py to print out errors that are new.
• Update collect-twitter-oembeds.py to include returned messages with "error" to be included in the "error" field.

Add a clear button to data viz interface

Experiment: place blur on background headers

Twitter screenshots useful?

Some of the media within tweets could be useful, and it might be a last-ditch way to get twitter-like tooltips.

Try alternating color schemes for each text section

.section:nth-child(odd)

Explore options to organize RJ media into graphic

"Themes of BLM media" might be an interesting section or sidenote. The data has already been collected for the header.

Reintroduce hyperlinks and styling

Write generate script for number of corps tweeting about RJ

Add footnotes of code snippets to each number / statement

This is not supported from GitHub's native markdown, but still important

Try to remove bootstrap

Some tooltip CSS is making things look nice, but it's not necessary.

Address errors in twitter oembeds

found in docs/twitter-oembeds.json with the key "errors"

Adapt scripts to new dataset structure

After stripping out the blm-tweets.csv and changing headers, current make scripts shouldn't work