Category: Software

Tuning text analytics for research students’ writing

UTS doctoral researcher Sophie Abel has just presented her latest work to the Ascilite conference, in a paper and talk entitled: Designing personalised, automated feedback to develop students’ research writing skills. — browse her talk slides below

Progress with the Text Analytics Pipeline (TAP)

The Text Analytics Pipeline (TAP) software has undergone significant improvements over the last few months. HETA project Technical Lead Andrew Gibson (QUT) has been working on a batch mode that allows the processing of unlimited numbers of documents from Amazon S3 Cloud storage, opening the way for more scalable analytics on millions of documents.

The TAP client for python (tapclipy) is also being improved to allow the batch mode to be used from within Python. The latest version of TAP can always be found at https://github.com/heta-io/tap and the python client at https://github.com/heta-io/tapclipy . If you’re looking to try bleeding-edge features, check out the develop branch, otherwise for a stable version use the master branch.

TAP has previously suffered from a lack of clear documentation (as do many software research projects). However, this issue is now being addressed by Mike Powell (QUT) together with Sarah Taylor (RMIT). The revised documentation can be found at https://heta-io.github.io/tap/ and includes instructions for getting started from Docker, or with the Source code, as well as helpful information for developers.

A new tool for navigating feedback from student surveys

One of the three priorities for the HETA project is to progress technology and analytics for extracting information and insight from student comments obtained from feedback surveys. In general, the “quantitative and categorical data” and the “free text comments” from student surveys are analysed independently. The first often in terms of aggregate statistics and the later simply read by interested parties. A joint analysis that merges the quantitative (scale-from-one-to-five type) responses and the richer specific information in the associated open-ended text responses allows a more powerful and nuanced extraction of information.

As a first step toward this more integrated approach, we have developed an online navigation tool for comments from the UTS Student Feedback Survey. This tool joins together, for the first time at UTS, the quantitative and free-text responses and provides an assortment of filtering and navigation utilities.  This tool brings together the last 4 years of student feedback on their experience in individual subjects and contains a total of over a quarter-of-a-million written student comments.

Screenshot of part of the filtered and sorted output from Student Feedback Comment Navigator. It provides the student comments for both the standard free-text-response survey questions juxtaposed with their quantitative responses. The results are listed and enumerated in order and can be scrolled through.  It also provides the functionality to sort the comments by the score given to any of the quantitative questions.
Screenshot of part of the filtered and sorted output from Student Feedback Comment Navigator. It provides the student comments for both the standard free-text-response survey questions juxtaposed with their quantitative responses. The results are listed and enumerated in order and can be scrolled through. It also provides the functionality to sort the comments by the score given to any of the quantitative questions.

Contextualizable learning analytics for writing support

UTS doctoral researcher Shibani Antonette gave a presentation today that will be of interest to HETA readers following our WP1 stream on text analytics for writing analysis. The key question she tackles is how we scale text analytics whilst also recognising the important contextual differences of students engaged in different kinds of writing, in different disciplines.

Here she is in action, full details with slides on her blog post…

Co-designing automated feedback on reflective writing with the teacher

Building on a previous co-design session, Ming Liu (writing analytics research fellow) and Simon Buckingham Shum (project lead) recently ran a follow-up session with Cherie Lucas (Discipline of Pharmacy, Graduate School of Health). The task was to design the first version of the Feedback Tab in AcaWriter, for reflective writing.

The AcaWriter screen looked like this:

Zooming in, the feedback looked like this (click to enlarge), with sentences annotated using icons and font:

(Learn more about the underlying model of textual features, and a study to evaluate initial student reactions to it.)

PhD work by Shibani Antonette has added a new Feedback Tab for other genres of student writing in Law (essays) and  Accounting (business analyses), while Sophie Abel has designed feedback for PhD students’ on their research abstracts. As you can see from those examples, in addition to the Analytical Report Tab which annotates sentences in the student’s text, the Feedback Tab gives explicit summaries about the meaning of the highlighting, and suggesting what the student might do to improve their draft. Here’s an example from Law:

So, this is what we needed to do for reflective writing. The task was to define a set of rules, which will trigger feedback advice to students given the presence or absence of particular features. The 2 hour design session was set up as shown below, with a Google Doc template on the left screen, and AcaWriter on the right:

For a given feature (col.1), we considered what should be said to the student if it appeared (col.2) or was missing (col.3). You can also see that more complex patterns emerged:

Presence of one feature but absence of another:

(triangle without square) While it appears that you’ve reported on how you would change/prepare for the future, you don’t seem to have described your thoughts, feelings and/or reactions to an incident, or learning task.

(triangle without preceding circle) While it appears that you’ve reported on how you would change/prepare for the future, you don’t seem to have reported first on what you found challenging. Perhaps you’ve reflected only on the positive aspects in your report?

Repeated feature in successive sentences:

(double circles) Well done, it appears that you may have expanded the detail on the challenge you faced.

(double triangles) Well done, it appears that you have expanded the detail on how you would change/prepare for the future.

Location-specific features:

(triangle in para1) It appears that you have reflected on this very early on. Please ensure that you recap this in your conclusion about the outcomes of your reflection.

Note the qualified tone of the feedback: it appears that you have… you don’t seem to have… Writing is so complex that the machine will undoubtedly get things wrong (something we’ve quantified – as one measure of quality). However, as we’ve argued elsewhere, it may be that imperfect analytics have specific uses for scaffolding higher order competencies in students.

After 2 hours, we had a completed template, which we could hand over to our developer to be implemented. The Feedback Tab is no longer empty…

Header on all feedback:

Encouraging feedback when features are present:

Cautionary feedback when features are absent:

To summarise, co-design means giving voice and influence to the relevant stakeholders in the design process. Too often, it feels to academics and teachers as though they’re doing all the adjusting to educational technology products, rather than being able to shape them. Since we have complete control over our writing analytics infrastructure (and so can you, since we’ve released it open source), academics can shape the functionality and user experience of the tool in profound ways.

Ultimately, we need to build infrastructure that educators and students trust, and there are many ways to tackle this, co-design being just one.

How do students respond to this automated feedback? Trials are now being planned… We’ll let you know!…

Writing analytics: online training in rhetorical parsing

As part of building training resources to upskill the teams in different kinds of text analytics, UTS:CIC’s Honorary Associate Dr Ágnes Sándor (Naver Labs Europe) has been running approx monthly online sessions. In these, Ágnes introduces the  concept matching model, and its rule-based implementation in the open source Athanor server. This powers AcaWriter’s ability to detect academic rhetorical moves, which in combination with other text analysis services in TAP, is being tuned and evaluated with student writing of different sorts.

Check out the replays and presentation and exercise slides.

TAP/AWA tutorial (Sydney, 5 March)

The team will be running a half-day tutorial on 5 March in Sydney, as part of the International Conference on Learning Analytics & Knowledge.

Turning the TAP  on Writing Analytics

Organisers: Antonette Shibani, Sophie Abel, Andrew Gibson and Simon Knight

Writing analytics is seen as a potentially useful technique that uses textual features to provide formative feedback on students’ writing. However, for this feedback to be effective, it is important that it is aligned to pedagogic contexts. Such efficient integration of technology in pedagogy could be supported by developing writing analytics literacy. The proposed workshop aims to build this capacity by mapping technical constructs to a broader educational sense for pragmatic applications. It provides a hands-on experience for participants to work with text analytics and discuss its implications for writing feedback. Participants will work with a set of text analysis code to extract features and map them to writing feedback. They will also be given the opportunity to develop rules based on extracted text features to write feedback for their own pedagogic contexts.

Writing Analytics R&D

The R&D program at UTS has developed and piloted a tool to provide automated formative feedback to students on their writing. The research publications below document how we’re designing this, and what we’re learning.

In addition, the team runs regular workshops and tutorials (2016/2017/2018) bringing together some of the world’s leading researchers to reflect on the state of the art and future of automated writing analysis. Continue reading “Writing Analytics R&D”