Easy tools for acquiring, processing and exploring data

This content is not yet complete. In the meantime, see this presentation: Easy tools for processing and exploring data (pdf, gd)

Acquiring data

• Introduction to web scraping (quite advanced, but contains a section on a user interface tool as well)

• Hand-written text transcription: Transkribus

• Layout and text transcription: OCR4all

• Keyword generation from text: Annif

• An automated sound transcription tool

• Twitter archiving: TAGS

Data processing

Assignment: data processing tools

• Complete the OpenRefine tutorial

Further resources

• OpenRefine for Social Science Data Data Carpentry tutorial, not really for social science but for general cleaning up of data

• Further tutorials:

  • http://enipedia.tudelft.nl/wiki/OpenRefine_Tutorial

  • http://j.mp/dhh15ho (includes section on extension)

  • http://freeyourmetadata.org/reconciliation/ (on reconciliation)

Data exploration

Visualisation

Visualisation is the act of taking data and transforming it into visual shapes and forms. The reasoning behind this is that humans are very good at processing visual information, with a lot of the necessary shape and anomaly detection and comparison processes even happening subconsciously.

Most visualisation is explanatory. https://pudding.cool/2017/05/song-repetition/

• http://www.datajourneyman.com/2016/03/21/preattentive-processing.html / https://medium.com/@vidya83.kesavan/preattentive-attributes-in-visualization-an-example-cb05ba1c9371

• Book: Information visualization : perception for design, particularly chapter 5 for pre-attentive processing and 6 for the gestalt laws.

• http://socviz.co/lookatdata.html section 1.5

Assignment

Read Perception deception & Common visualization mistakes. The articles are primarily oriented around explanatory data visualisation, but most computational humanities data analysis is exploratory. How do the problems transfer into that domain, when the only one you can deceive is yourself?

Assignment: data exploration tools

Experiment with at least one of the following tools:

• tabular data → chart visualisations: RAW​

• tabular data → chart visualisations: Voyager

• tabular data → chart visualisations: ​Tableau​

• tabular data → ​interactive map/network/timeline/list/facet visualisations: Palladio​

  • Palladio has help pages. There are also multiple tutorials on using Palladio, for example this one, or this one which is particularly on network analysis.

• tabular data → map(+timeline) visualisations: ​Carto​

• ​text →​ interactive explorative interface for linguistic study: Voyant tools​

• ​big, preselected collections of text → interface for linguistic study: Korp / corpus.byu.edu​

• If you're feeling explorative, feel free to also dig for more tools in TAPoR.

If you're short on inspiration, feel free to go through this hands-on tutorial covering OpenRefine, RAW and Palladio.

Afterwards, post a message on #tools on Slack detailing:

1. What is the tool good for?

2. What kind of data do you need for the tool to be useful?

   1. What information does the data need to contain?

   2. What format does it have to be in?

3. Your experience with the tool.

If someone has already posted on the tool you tested, don't repeat them. Instead, add to what they've said in a thread.

Assignment: visualisation tool development

Read the following two research articles on developing visualisation tools for particular text-based humanities research questions:

• Visualizing Mouvance: Toward a visual analysis of variant medieval text traditions

• Rule‐based Visual Mappings – with a Case Study on Poetry Visualization

Now, think of a visualisation that would help you in your field. What information would it visualise? Post a message on #tools on Slack

Resources

• Flowchart on selecting a good visualization based on what you want to show

• An example of four ways to visualise the same data and how that affects what you can read from it