SI16 Backend

Familiar and flexible

Each contribuition to the SI16 API was unique. Both the subgroups' projects and their documentation had different voices, different ways of presenting the contents, and different needs. This specificity required a system that was structured enough to keep together each pace, but remained flexible, in order to let anyone express her own approach to the Special Issue.

With the SI16 being at the same time a set of functions, a playground to experiment with it, and a list of meaningful projects developed within their context, we structured the backend as an interface between the different parts of the work.

The main idea was to define a pipeline that was not so different from the processes and technologies we learned and explored during the first trimester.

Functions

We chose to work with the Jupiter Notebook format we were familiar with, as a way to both collect and document our functions. With a common protocol defined within us and some 🐍 pythonic 🐍 (omg this term is cringe ah ah) good practices, we wrote a Notebook for each function. In each file there was the definition of the function, as well as a propper documentation and some examples on how to use it.

A big part of the backend work was to let anyone structure their own notebook freely, deciding how to present the process and the result of each contribuition without forcing a structure. At the end the Flask application was designed to scan all the notebook folder and extract from it the function, as well as their input and output, and the documentation.

With those information we generated an interactive page for each function in which the user could try and play around with our functions exposed as an API.

Projects

The organic process of SI16 led us to a collection of several interconnected projects. Each one of them had grown around a specific implementation of the functions we developed for the API. Those complex applications were initally developed as standalone Flask app, and they were merged all together at the end. (well, ok, after the end to be honest)

One thing we could do differently next time is to use Flask's Blueprints as a way to work in a more flexible way. At some point we were really struggling about how to manage the code and the collaboration on it. Now that we have an overview of how Python works, it could be nice to develop our projects in a more modular way. BTW we used the documentation pages as a gateway to the projects, in order to have a common starting point.

Things got a bit complicated when the subgroup I was in started to working on sub-sub-project. And to face that we made the structure open to the possibility to nest projects one into the other, in order to have again flexibility between documentation and interaction on the website, and still a (kinda) structure in the filesystem.

Actually: we were totally new into this, so probably half of the things we did were not just wrong, but illegal. BTW we are still learning so next time it will be better, deal with it.

disclaimer there are some problems with the xpub git so the source code is still not totally public, but we are working on it. There the code is annotated and each function is documented so I will not go more in detail here for now.

Project structure

si16
├── contents
├── notebooks
├── projects
│   ├── and-i-wish-that-your-question-has-been-answered
│   ├── annotation-compass
│   │   └── showcases
│   └── etc-portal
├── si16.py
├── static
│   ├── corpora
│   ├── css
│   ├── event
│   ├── font
│   ├── img
│   ├── js
│   └── uploads
└── templates

• contents contains a list of markdown file for generic pages. For example the route /si16/intro will search and load the contents from the intro.md file in this folder.
• notebooks contains a list of Jupiter Notebook files, one for each function we developed for the API.
• projects contains a list of folder one for each subgroup project. Each one has a documentation.md file with the main contents and process of development.
• showcases is a optional folder in each project for nesting other sub projects in the sub project. Convoluted but useful and flexible.
• si16.py is the main module for the Flask app and the backend duties.
• static contains all the different files that are served statically from the webserver such as images, stylesheets, javascript files, etc.
• templates is a Flask folder that contains the HTML templates for generating the frontend pages.

Ciao ciao